Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 917907 Details for
Bug 1119376
Warning about OVF disk when creating a domain
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
vdsm-log-3-4
vdsm.log (text/x-log), 10.11 MB, created by
Gilad Chaplik
on 2014-07-14 16:03:17 UTC
(
hide
)
Description:
vdsm-log-3-4
Filename:
MIME Type:
Creator:
Gilad Chaplik
Created:
2014-07-14 16:03:17 UTC
Size:
10.11 MB
patch
obsolete
>Dummy-49::DEBUG::2014-07-14 13:01:01,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:01,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923862 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:01:03,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:01:03,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:01:03,066::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00105881 s, 350 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:01:03,069::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00179703 s, 291 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:03,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:03,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092335 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:01:03,971::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:01:05,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:05,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913393 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:01:06,296::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:01:06,297::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:01:07,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:08,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913769 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:10,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:01:10,581::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:01:10,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503267 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:12,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:12,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916081 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:01:13,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:01:13,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:01:13,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000752408 s, 493 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:01:13,096::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000717866 s, 729 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:14,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:14,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914068 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:01:16,313::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:01:16,317::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:01:16,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:18,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37684 s, 744 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:01:18,985::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:01:20,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:20,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912965 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:22,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:22,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922503 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:01:23,100::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:01:23,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000820222 s, 452 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:01:23,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:01:23,128::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000601152 s, 870 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:24,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:24,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912015 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:01:26,324::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:01:26,325::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:01:26,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:26,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912754 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:28,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:29,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.512529 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:31,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:31,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0929644 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:01:33,130::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:01:33,142::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:01:33,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000732913 s, 506 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:01:33,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700594 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:33,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:33,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0980327 s, 10.4 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:01:33,996::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:01:35,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:35,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915509 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:01:36,336::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:01:36,337::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:01:37,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:37,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920825 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:39,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:40,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54974 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:01:40,613::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:01:42,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:42,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912157 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:01:43,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:01:43,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:01:43,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00500674 s, 104 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:01:43,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632274 s, 587 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:44,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:44,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912349 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:01:46,354::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:01:46,357::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:01:46,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:46,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911369 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:48,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:48,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091222 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:01:48,999::task::595::Storage.TaskManager.Task::(_updateState) Task=`45570d2b-f838-4f89-958b-211c9ab0c21d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:01:49,000::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:01:49,004::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:01:49,004::task::1191::Storage.TaskManager.Task::(prepare) Task=`45570d2b-f838-4f89-958b-211c9ab0c21d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:01:49,004::task::595::Storage.TaskManager.Task::(_updateState) Task=`45570d2b-f838-4f89-958b-211c9ab0c21d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:01:49,004::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:01:49,004::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:01:49,005::task::993::Storage.TaskManager.Task::(_decref) Task=`45570d2b-f838-4f89-958b-211c9ab0c21d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:01:49,031::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:01:50,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:50,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911801 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:52,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:53,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912713 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:01:53,206::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 13:01:53,215::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:01:53,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689655 s, 538 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:01:53,240::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:01:53,241::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:01:53,244::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:01:53,246::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:01:53,246::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:01:53,246::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:01:53,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:01:53,258::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065426 s, 799 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:55,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:55,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912576 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:01:56,364::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:01:56,365::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:01:57,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:57,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912433 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:01:59,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:01:59,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913151 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:01,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:01,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546918 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:02:03,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:02:03,252::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622783 s, 596 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:02:03,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:02:03,274::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630668 s, 829 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:03,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:02:04,043::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:02:04,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091103 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:06,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:06,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912578 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:02:06,371::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:02:06,373::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:02:08,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:08,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912422 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:10,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:02:10,646::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:02:10,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555163 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:12,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:12,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911243 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:02:13,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:02:13,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656885 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:02:13,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:02:13,289::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700423 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:14,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:15,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915991 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:02:16,389::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:02:16,394::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:02:17,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:17,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091206 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:02:19,055::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:02:19,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:19,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550183 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:21,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:21,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914161 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:02:23,269::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:02:23,282::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:02:23,283::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:02:23,286::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:02:23,288::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:02:23,288::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:02:23,288::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:02:23,289::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:02:23,301::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:02:23,312::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0042885 s, 86.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:02:23,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696877 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:23,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:24,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912886 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:26,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:26,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912638 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:02:26,394::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:02:26,395::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:02:28,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:28,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914715 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:30,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:30,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502593 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:32,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:32,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911984 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:02:33,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:02:33,329::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:02:33,340::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00526589 s, 70.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:02:33,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700202 s, 747 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:02:34,067::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:02:34,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:35,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.571154 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:02:36,407::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:02:36,408::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:02:37,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:38,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542184 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:40,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:40,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913681 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:02:40,678::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:02:42,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:42,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912674 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:02:43,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:02:43,357::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:02:43,368::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690374 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:02:43,370::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000955895 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:44,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:44,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911859 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:02:46,421::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:02:46,422::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:02:46,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:46,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0929418 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:48,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:48,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913744 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:02:49,070::task::595::Storage.TaskManager.Task::(_updateState) Task=`6abd101f-41f8-4864-826a-96a6b2571ce5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:02:49,070::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:02:49,073::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:02:49,074::task::1191::Storage.TaskManager.Task::(prepare) Task=`6abd101f-41f8-4864-826a-96a6b2571ce5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:02:49,074::task::595::Storage.TaskManager.Task::(_updateState) Task=`6abd101f-41f8-4864-826a-96a6b2571ce5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:02:49,074::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:02:49,074::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:02:49,074::task::993::Storage.TaskManager.Task::(_decref) Task=`6abd101f-41f8-4864-826a-96a6b2571ce5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:02:49,102::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:02:50,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:50,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926522 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:52,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:52,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920189 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:02:53,374::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:02:53,386::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628373 s, 590 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:02:53,389::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:02:53,402::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000751686 s, 696 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:54,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:55,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911487 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:02:56,432::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:02:56,434::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:02:57,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:57,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916698 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:02:59,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:02:59,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559897 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:01,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:03,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.3987 s, 732 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:03:03,651::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:03:03,663::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:03:03,673::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632951 s, 826 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:03:03,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674527 s, 550 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:03:04,114::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:03:05,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:05,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091173 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:03:06,445::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:03:06,446::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:03:07,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:07,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091329 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:09,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:09,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912606 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:03:10,707::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:03:11,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:12,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37329 s, 746 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:03:13,681::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:03:13,692::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:03:13,703::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0044766 s, 82.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:03:13,705::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00109386 s, 478 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:14,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:15,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911955 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:03:16,464::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:03:16,467::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:03:17,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:17,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914256 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:03:19,126::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:03:19,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:19,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912377 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:21,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:21,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911024 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:23,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:23,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912249 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:03:23,707::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:03:23,719::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00062429 s, 594 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:03:23,721::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:03:23,734::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662848 s, 789 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:25,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:25,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911867 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:03:26,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:03:26,475::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:03:27,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:28,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556943 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:30,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:30,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913452 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:32,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:32,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091153 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:03:33,724::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:03:33,736::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704963 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:03:33,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:03:33,750::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068986 s, 758 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:03:34,138::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:03:34,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:34,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.553934 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:03:36,486::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:03:36,487::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:03:37,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:37,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914569 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:39,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:39,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912767 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:03:40,742::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:03:41,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:42,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37486 s, 745 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:03:43,739::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:03:43,751::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652214 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:03:43,755::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:03:43,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643243 s, 813 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:44,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:44,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091309 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:03:46,501::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:03:46,504::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:03:46,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:47,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545335 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:03:49,147::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9a7f3be-b5e6-4f8b-8168-25b2ea550412`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:03:49,147::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:03:49,150::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:03:49,151::task::1191::Storage.TaskManager.Task::(prepare) Task=`e9a7f3be-b5e6-4f8b-8168-25b2ea550412`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:03:49,151::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9a7f3be-b5e6-4f8b-8168-25b2ea550412`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:03:49,151::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:03:49,151::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:03:49,151::task::993::Storage.TaskManager.Task::(_decref) Task=`e9a7f3be-b5e6-4f8b-8168-25b2ea550412`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:03:49,176::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:03:49,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:49,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911895 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:51,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:52,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550477 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:03:53,756::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:03:53,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649772 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:03:53,770::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:03:53,782::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694326 s, 753 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:54,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:54,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549792 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:03:56,512::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:03:56,514::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:03:56,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:56,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0930999 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:03:58,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:03:58,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913468 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:00,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:00,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911492 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:02,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:03,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0977194 s, 10.5 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:04:03,772::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:04:03,785::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663083 s, 560 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:04:03,787::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:04:03,801::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631348 s, 828 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:04:04,187::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:04:05,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:05,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911964 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:04:06,521::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:04:06,522::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:04:07,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:07,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911707 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:09,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:09,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912631 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:04:10,789::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:04:11,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:11,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913816 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:13,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:13,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913482 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:04:13,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:04:13,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000710143 s, 522 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:04:13,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:04:13,830::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686305 s, 762 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:15,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:15,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0963022 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:04:16,538::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:04:16,539::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:04:17,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:18,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911155 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:04:19,200::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:04:20,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:20,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911791 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:22,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:22,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912636 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:04:23,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:04:23,844::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000747102 s, 497 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:04:23,844::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:04:23,857::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675487 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:24,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:24,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911756 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:26,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:26,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091418 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:04:26,556::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:04:26,562::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:04:28,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:28,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912146 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:30,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:30,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912193 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:32,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:32,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091201 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:04:33,859::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:04:33,870::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:04:33,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685617 s, 541 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:04:33,887::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671764 s, 779 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:04:34,212::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:04:34,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:35,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091259 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:04:36,564::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:04:36,565::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:04:37,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:37,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912761 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:39,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:39,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927619 s, 11.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:04:40,821::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:04:41,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:41,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091297 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:43,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:04:43,908::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:04:43,919::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:04:43,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690104 s, 758 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:04:43,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625393 s, 593 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:43,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551708 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:45,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:46,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0979006 s, 10.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:04:46,576::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:04:46,577::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:04:48,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:48,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913451 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:04:49,215::task::595::Storage.TaskManager.Task::(_updateState) Task=`6ca9e6c3-1516-4cb8-b754-57b5db995108`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:04:49,216::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:04:49,219::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:04:49,219::task::1191::Storage.TaskManager.Task::(prepare) Task=`6ca9e6c3-1516-4cb8-b754-57b5db995108`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:04:49,219::task::595::Storage.TaskManager.Task::(_updateState) Task=`6ca9e6c3-1516-4cb8-b754-57b5db995108`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:04:49,219::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:04:49,219::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:04:49,220::task::993::Storage.TaskManager.Task::(_decref) Task=`6ca9e6c3-1516-4cb8-b754-57b5db995108`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:04:49,248::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:04:50,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:50,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922308 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:52,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:53,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39301 s, 735 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:04:53,949::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:04:53,960::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:04:53,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00589413 s, 62.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:04:53,973::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00110679 s, 473 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:04:55,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:55,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091303 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:04:56,589::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:04:56,590::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:04:57,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:04:58,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557646 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:00,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:00,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913311 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:02,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:02,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912978 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:05:03,977::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:05:03,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:05:03,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000634948 s, 584 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:05:04,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641543 s, 815 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:05:04,261::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:05:04,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:05,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544726 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:05:06,601::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:05:06,602::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:05:07,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:07,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913348 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:09,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:09,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913305 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:05:10,855::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:05:11,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:11,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912103 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:13,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:13,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920341 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:05:14,004::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:05:14,015::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:05:14,027::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00526935 s, 70.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:05:14,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652422 s, 802 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:15,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:15,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967585 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:05:16,611::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:05:16,612::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:05:18,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:18,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912688 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:05:19,282::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:05:20,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:20,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914685 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:22,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:22,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913042 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:05:24,391::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:05:24,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:05:24,417::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620797 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:05:24,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649947 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:24,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:24,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912775 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:26,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:05:26,624::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:05:26,625::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:05:26,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912645 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:28,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:28,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912089 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:30,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:30,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912292 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:32,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:33,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912926 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:05:34,294::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:05:34,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:05:34,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:05:34,446::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620166 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:05:34,447::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689008 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:35,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:35,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913608 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:05:36,636::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:05:36,637::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:05:37,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:37,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132994 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:39,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:39,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917408 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:05:40,888::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:05:41,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:41,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551724 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:44,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:05:44,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:05:44,571::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:05:44,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629988 s, 589 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:05:44,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000624649 s, 837 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:44,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.581183 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:46,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:05:46,646::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:05:46,648::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:05:47,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.365 s, 750 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:05:49,298::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c61997c-4c1c-4e3b-a404-0a01f1bee49a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:05:49,298::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:05:49,762::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:05:49,762::task::1191::Storage.TaskManager.Task::(prepare) Task=`3c61997c-4c1c-4e3b-a404-0a01f1bee49a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:05:49,763::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c61997c-4c1c-4e3b-a404-0a01f1bee49a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:05:49,763::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:05:49,763::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:05:49,763::task::993::Storage.TaskManager.Task::(_decref) Task=`3c61997c-4c1c-4e3b-a404-0a01f1bee49a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:05:49,789::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:05:50,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:50,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912865 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:52,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:52,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912003 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:54,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:05:54,789::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:05:54,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:05:54,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666169 s, 785 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:05:54,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687424 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:54,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.578486 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:05:56,658::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:05:56,660::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:05:56,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:56,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911972 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:05:58,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:05:59,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911502 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:01,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:01,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912761 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:03,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:03,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091246 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:06:04,800::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 13:06:04,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:06:04,837::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:06:04,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720218 s, 726 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:06:04,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000929604 s, 399 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:05,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:05,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911586 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:06:06,670::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:06:06,671::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:06:07,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:07,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912644 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:09,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:09,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913094 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:06:10,922::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:06:11,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:11,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912619 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:13,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:13,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912682 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:06:14,853::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:06:14,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:06:14,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00242391 s, 216 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:06:14,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652307 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:15,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:16,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0998335 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:06:16,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:06:16,683::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:06:18,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:18,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912532 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:06:19,813::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:06:20,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:20,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911867 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:22,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:22,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091228 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:24,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:24,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912809 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:06:24,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:06:24,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:06:24,907::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00154707 s, 338 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:06:24,909::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00100387 s, 370 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:26,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:06:26,694::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:06:26,696::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:06:27,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912641 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:29,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:30,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911592 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:32,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:32,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091519 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:34,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:34,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913096 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:06:34,824::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 13:06:34,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:06:34,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:06:34,937::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00252545 s, 207 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:06:34,939::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00119854 s, 310 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:36,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:36,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912116 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:06:36,707::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:06:36,708::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:06:38,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:38,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916715 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:40,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:06:40,954::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:06:41,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556546 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:43,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:43,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911338 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:06:44,947::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:06:44,957::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:06:44,968::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671072 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:06:44,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698191 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:45,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:45,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912889 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:06:46,716::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:06:46,717::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:06:47,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:47,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911534 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:49,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:49,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913444 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:06:49,827::task::595::Storage.TaskManager.Task::(_updateState) Task=`cee7ca19-7373-44ce-9874-bc07f70bdc23`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:06:49,828::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:06:49,830::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:06:49,830::task::1191::Storage.TaskManager.Task::(prepare) Task=`cee7ca19-7373-44ce-9874-bc07f70bdc23`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:06:49,831::task::595::Storage.TaskManager.Task::(_updateState) Task=`cee7ca19-7373-44ce-9874-bc07f70bdc23`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:06:49,831::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:06:49,831::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:06:49,831::task::993::Storage.TaskManager.Task::(_decref) Task=`cee7ca19-7373-44ce-9874-bc07f70bdc23`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:06:49,858::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:06:51,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:51,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911425 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:53,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:53,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912803 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:06:54,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:06:54,982::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 13:06:54,986::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000707806 s, 524 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:06:55,006::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:06:55,007::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:06:55,010::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:06:55,011::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:06:55,012::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:06:55,012::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:06:55,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:06:55,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067709 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:06:55,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:55,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911988 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:06:56,726::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:06:56,727::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:06:57,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:06:57,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912644 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:00,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:00,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914608 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:02,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:02,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912591 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:04,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:04,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912443 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:07:04,870::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:07:05,000::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:07:05,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618815 s, 600 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:07:05,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:07:05,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642299 s, 814 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:06,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:06,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925728 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:07:06,731::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:07:06,733::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:07:08,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:08,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091214 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:10,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:10,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913843 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:07:11,001::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:07:12,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:12,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922597 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:14,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:14,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912555 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:07:15,016::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:07:15,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000634036 s, 585 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:07:15,046::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:07:15,057::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663881 s, 788 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:07:16,740::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:07:16,742::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:07:16,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:17,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913449 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:19,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:19,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914801 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:07:19,883::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:07:21,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:21,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912087 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:23,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:23,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549406 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:07:25,031::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:07:25,045::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:07:25,047::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:07:25,050::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:07:25,052::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:07:25,052::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:07:25,052::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:07:25,054::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:07:25,067::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665113 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:07:25,069::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:07:25,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694032 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:25,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:25,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912196 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:07:26,753::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:07:26,754::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:07:27,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:28,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911632 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:30,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:30,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913438 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:32,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:32,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913669 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:34,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:34,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912548 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:07:34,896::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:07:35,086::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:07:35,098::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000742107 s, 500 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:07:35,099::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:07:35,112::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000781464 s, 669 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:36,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:36,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913284 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:07:36,761::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:07:36,762::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:07:38,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:38,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914454 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:40,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:40,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914367 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:07:41,032::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:07:42,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:42,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:44,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:45,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913021 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:07:45,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:07:45,127::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000722705 s, 513 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:07:45,130::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:07:45,143::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669925 s, 781 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:07:46,770::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:07:46,772::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:07:47,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:47,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091296 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:49,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:49,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911165 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:07:49,899::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb2fe0b1-d705-4be7-9b97-c10eb2a6356d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:07:49,900::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:07:49,902::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:07:49,902::task::1191::Storage.TaskManager.Task::(prepare) Task=`bb2fe0b1-d705-4be7-9b97-c10eb2a6356d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:07:49,902::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb2fe0b1-d705-4be7-9b97-c10eb2a6356d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:07:49,903::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:07:49,903::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:07:49,903::task::993::Storage.TaskManager.Task::(_decref) Task=`bb2fe0b1-d705-4be7-9b97-c10eb2a6356d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:07:49,930::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:07:51,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:51,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912475 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:53,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:53,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091227 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:07:55,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:07:55,158::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691514 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:07:55,160::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:07:55,173::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657032 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:55,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:55,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091316 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:07:56,791::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:07:56,795::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:07:57,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:57,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912264 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:07:59,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:07:59,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914543 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:01,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:03,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39952 s, 732 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:08:04,943::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:08:05,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:08:05,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:08:05,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000638302 s, 581 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:08:05,200::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707866 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:05,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:05,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921641 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:08:06,791::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:08:06,792::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:08:07,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:07,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913932 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:09,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:09,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912831 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:08:11,065::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:08:11,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:11,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091153 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:13,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:13,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913644 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:08:15,205::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:08:15,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:08:15,227::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00605108 s, 61.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:08:15,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068764 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:15,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:15,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0998308 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:08:16,803::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:08:16,804::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:08:18,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:18,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919482 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:08:19,956::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:08:20,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:20,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548001 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:22,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:23,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911979 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:25,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:25,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915848 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:08:25,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:08:25,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:08:25,256::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00559427 s, 66.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:08:25,257::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642662 s, 814 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:08:26,815::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:08:26,816::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:08:27,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:27,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912696 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:29,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:29,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911755 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:31,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:31,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913363 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:33,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:33,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912138 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:08:34,968::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:08:35,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:08:35,273::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:08:35,284::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000719707 s, 515 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:08:35,285::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000726113 s, 720 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:35,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:35,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913386 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:08:36,827::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:08:36,829::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:08:37,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:38,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503301 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:40,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:40,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913434 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:08:41,096::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:08:42,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:42,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.533926 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:44,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:45,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922873 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:08:45,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:08:45,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:08:45,304::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00064364 s, 576 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:08:45,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065342 s, 800 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:08:46,837::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:08:46,838::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:08:47,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:47,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912686 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:49,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:49,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911344 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:08:49,971::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ce24a5e-b650-484a-b192-93359a5e5a39`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:08:49,971::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:08:49,974::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:08:49,975::task::1191::Storage.TaskManager.Task::(prepare) Task=`7ce24a5e-b650-484a-b192-93359a5e5a39`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:08:49,975::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ce24a5e-b650-484a-b192-93359a5e5a39`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:08:49,975::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:08:49,975::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:08:49,975::task::993::Storage.TaskManager.Task::(_decref) Task=`7ce24a5e-b650-484a-b192-93359a5e5a39`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:08:50,003::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:08:51,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:51,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911781 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:53,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:53,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910959 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:08:55,320::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:08:55,333::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:08:55,334::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659345 s, 563 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:08:55,347::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000801638 s, 652 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:08:55,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:55,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912244 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:08:56,849::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:08:56,850::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:08:57,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:08:58,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559025 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:00,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:00,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912119 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:02,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:02,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912802 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:04,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:09:05,015::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:09:05,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561096 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:09:05,352::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:09:05,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:09:05,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000708296 s, 524 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:09:05,382::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000769738 s, 679 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:09:06,858::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:09:06,860::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:09:07,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:07,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914773 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:09,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:09,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912375 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:09:11,128::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:09:11,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:11,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912459 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:13,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:13,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911953 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:09:15,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:09:15,397::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:09:15,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000702005 s, 528 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:09:15,410::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687853 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:15,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:16,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555376 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:09:16,871::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:09:16,872::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:09:18,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:18,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912799 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:09:20,028::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:09:20,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:20,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912491 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:22,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:22,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.094667 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:24,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:24,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911926 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:09:25,415::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:09:25,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:09:25,437::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651104 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:09:25,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640006 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:26,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:26,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911568 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:09:26,883::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:09:26,884::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:09:28,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:28,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911951 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:30,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:31,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919821 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:33,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:33,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913599 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:09:35,039::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:09:35,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:35,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911643 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:09:35,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:09:35,452::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:09:35,467::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000719494 s, 516 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:09:35,469::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670975 s, 779 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:09:36,893::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:09:36,894::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:09:37,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:37,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913791 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:39,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:39,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919979 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:09:41,161::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:09:41,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:41,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911771 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:43,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:44,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544135 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:09:45,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:09:45,484::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:09:45,500::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000588623 s, 630 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:09:45,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720332 s, 726 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:46,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:46,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0997938 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:09:46,906::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:09:46,907::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:09:48,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:49,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502271 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:09:50,042::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a3678cb-8698-4e91-ae9a-84250b29e212`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:09:50,043::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:09:50,046::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:09:50,046::task::1191::Storage.TaskManager.Task::(prepare) Task=`3a3678cb-8698-4e91-ae9a-84250b29e212`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:09:50,046::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a3678cb-8698-4e91-ae9a-84250b29e212`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:09:50,046::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:09:50,046::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:09:50,047::task::993::Storage.TaskManager.Task::(_decref) Task=`3a3678cb-8698-4e91-ae9a-84250b29e212`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:09:50,073::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:09:51,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:51,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912683 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:53,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:53,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912969 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:55,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:55,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912611 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:09:55,555::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:09:55,555::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:09:55,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00198095 s, 264 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:09:55,578::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00062176 s, 597 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:09:56,914::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:09:56,915::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:09:57,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:09:57,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550401 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:09:59,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:00,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545195 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:02,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:02,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913251 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:04,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:04,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912492 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:10:05,087::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 13:10:05,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:10:05,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:10:05,606::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683616 s, 765 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:10:05,608::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000640317 s, 579 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:06,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:06,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911855 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:10:06,924::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:10:06,925::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:10:08,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:09,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542948 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:10:11,194::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:10:11,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:11,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911541 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:13,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:13,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912952 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:10:15,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:10:15,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:10:15,643::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066745 s, 784 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:10:15,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647988 s, 573 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:15,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:15,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922866 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:10:16,936::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:10:16,937::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:10:17,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:18,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510181 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:10:20,101::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:10:20,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:20,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912214 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:22,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:23,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547918 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:25,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:25,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913128 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:10:25,656::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:10:25,667::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:10:25,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00546731 s, 67.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:10:25,680::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062425 s, 838 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:10:26,948::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:10:26,949::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:10:27,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:27,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912135 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:29,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:29,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911154 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:31,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:31,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912717 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:33,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:33,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913728 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:10:35,113::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:10:35,685::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:10:35,686::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:10:35,707::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00269177 s, 138 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:10:35,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000879873 s, 594 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:35,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:35,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912529 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:10:36,960::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:10:36,962::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:10:37,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:38,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915837 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:40,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:40,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912388 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:10:41,230::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:10:42,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:42,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911647 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:44,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:44,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544108 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:10:45,713::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:10:45,726::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:10:45,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680104 s, 546 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:10:45,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685069 s, 763 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:46,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:10:46,968::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:10:46,969::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:10:47,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543136 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:49,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:49,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091308 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:10:50,116::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b86581d-b6f4-4c6e-b56a-51f7f7026310`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:10:50,117::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:10:50,120::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:10:50,120::task::1191::Storage.TaskManager.Task::(prepare) Task=`0b86581d-b6f4-4c6e-b56a-51f7f7026310`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:10:50,120::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b86581d-b6f4-4c6e-b56a-51f7f7026310`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:10:50,121::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:10:50,121::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:10:50,121::task::993::Storage.TaskManager.Task::(_decref) Task=`0b86581d-b6f4-4c6e-b56a-51f7f7026310`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:10:50,149::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:10:51,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:52,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557494 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:54,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:54,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55253 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:10:55,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:10:55,759::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:10:55,770::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00543892 s, 68.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:10:55,772::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0010832 s, 483 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:10:56,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:56,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912793 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:10:56,978::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:10:56,979::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:10:58,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:10:58,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912688 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:00,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:01,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913048 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:03,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:03,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914528 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:05,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:11:05,166::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:11:05,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.500137 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:11:05,775::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:11:05,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:11:05,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000695577 s, 533 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:11:05,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661763 s, 790 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:11:06,986::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:11:06,987::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:11:07,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:07,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912072 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:09,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:09,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914995 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:11:11,262::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:11:11,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:12,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912011 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:14,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:14,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912842 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:11:15,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:11:15,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:11:15,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069774 s, 750 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:11:15,836::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669844 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:16,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:16,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0966262 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:11:16,997::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:11:16,998::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:11:18,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:18,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912829 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:11:20,180::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:11:20,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:20,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911994 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:22,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:22,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911171 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:24,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:24,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091089 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:11:25,844::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:11:25,855::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:11:25,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000761927 s, 487 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:11:25,867::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641032 s, 816 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:26,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:26,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912731 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:11:27,007::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:11:27,008::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:11:28,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:29,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924163 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:31,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:31,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091436 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:33,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:33,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912848 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:11:35,191::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:11:35,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:35,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912673 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:11:35,870::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:11:35,883::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063545 s, 584 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:11:35,884::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:11:35,896::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000613483 s, 853 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:11:37,019::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:11:37,020::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:11:37,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:37,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913989 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:39,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:39,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911919 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:11:41,295::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:11:41,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:41,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912141 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:44,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:44,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917907 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:11:45,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:11:45,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:11:45,927::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681147 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:11:45,930::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674389 s, 776 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:46,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:46,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.131204 s, 7.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:11:47,028::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:11:47,029::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:11:48,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:48,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912568 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:11:50,194::task::595::Storage.TaskManager.Task::(_updateState) Task=`203303b9-a997-4a58-b756-fb30b3a83fb1`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:11:50,195::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:11:50,198::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:11:50,198::task::1191::Storage.TaskManager.Task::(prepare) Task=`203303b9-a997-4a58-b756-fb30b3a83fb1`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:11:50,198::task::595::Storage.TaskManager.Task::(_updateState) Task=`203303b9-a997-4a58-b756-fb30b3a83fb1`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:11:50,198::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:11:50,198::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:11:50,199::task::993::Storage.TaskManager.Task::(_decref) Task=`203303b9-a997-4a58-b756-fb30b3a83fb1`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:11:50,224::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:11:50,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:50,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920601 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:52,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:52,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913654 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:54,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:54,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912368 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:11:55,937::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:11:55,948::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 13:11:55,952::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000746332 s, 497 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:11:55,968::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:11:55,970::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:11:55,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:11:55,974::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:11:55,974::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:11:55,975::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:11:55,975::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:11:55,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688825 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:11:56,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:56,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912192 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:11:57,041::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:11:57,042::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:11:58,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:11:59,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543559 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:01,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:01,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913429 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:03,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:03,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912506 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:12:05,236::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:12:05,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:05,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913189 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:12:05,963::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:12:05,976::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000634578 s, 585 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:12:05,991::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:12:06,003::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069133 s, 757 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:12:07,053::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:12:07,054::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:12:07,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:08,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913191 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:10,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:10,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912927 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:12:11,328::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:12:12,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:12,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912958 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:14,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:14,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911652 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:12:15,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:12:15,993::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697604 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:12:16,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:12:16,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000795861 s, 657 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:16,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:16,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555313 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:12:17,063::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:12:17,065::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:12:18,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:19,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544903 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:12:20,252::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:12:21,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:22,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545536 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:24,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:24,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091333 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:12:25,998::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:12:26,012::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:12:26,014::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:12:26,017::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:12:26,019::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:12:26,019::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:12:26,019::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:12:26,020::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:12:26,032::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:12:26,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00524451 s, 70.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:12:26,044::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000797013 s, 656 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:26,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:26,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552574 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:12:27,075::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:12:27,077::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:12:28,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:28,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911566 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:30,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:31,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914128 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:33,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:33,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913527 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:35,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:12:35,266::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:12:35,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911792 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:12:36,048::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:12:36,060::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:12:36,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00597593 s, 62.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:12:36,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00113543 s, 461 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:12:37,087::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:12:37,090::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:12:37,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:37,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914082 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:39,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:39,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912202 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:12:41,362::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:12:41,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:41,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091105 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:43,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:43,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091362 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:45,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:45,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0965878 s, 10.6 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:12:46,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:12:46,096::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:12:46,098::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688705 s, 539 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:12:46,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000796847 s, 656 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:12:47,099::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:12:47,101::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:12:48,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:48,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913078 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:50,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:50,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913555 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:12:50,270::task::595::Storage.TaskManager.Task::(_updateState) Task=`7595cd91-597e-47b6-8b67-a1b286040ccb`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:12:50,270::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:12:50,273::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:12:50,273::task::1191::Storage.TaskManager.Task::(prepare) Task=`7595cd91-597e-47b6-8b67-a1b286040ccb`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:12:50,273::task::595::Storage.TaskManager.Task::(_updateState) Task=`7595cd91-597e-47b6-8b67-a1b286040ccb`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:12:50,273::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:12:50,273::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:12:50,274::task::993::Storage.TaskManager.Task::(_decref) Task=`7595cd91-597e-47b6-8b67-a1b286040ccb`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:12:50,302::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:12:52,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:52,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912007 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:54,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:54,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911652 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:12:56,123::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:12:56,136::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626379 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:12:56,139::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:12:56,151::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692181 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:12:56,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:56,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912073 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:12:57,112::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:12:57,113::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:12:58,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:12:59,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543345 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:01,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:01,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913687 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:03,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:03,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910919 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:13:05,315::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:13:05,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:05,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913151 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:13:06,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:13:06,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:13:06,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000732068 s, 507 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:13:06,179::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668617 s, 782 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:13:07,121::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:13:07,122::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:13:07,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:07,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912066 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:09,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:09,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0937887 s, 10.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:13:11,400::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:13:11,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:11,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912063 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:13,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:14,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913147 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:16,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:13:16,590::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:13:16,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:13:16,605::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628641 s, 590 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:13:16,620::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000744365 s, 703 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:16,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.598222 s, 1.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:13:17,131::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:13:17,132::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:13:18,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:18,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912404 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:13:20,327::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:13:20,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:21,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546063 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:23,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:25,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.4591 s, 702 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:13:26,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:13:26,638::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:13:26,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000792024 s, 468 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:13:26,650::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656114 s, 797 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:13:27,144::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:13:27,145::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:13:27,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:27,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091185 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:29,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:29,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912203 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:31,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:31,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091307 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:33,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:34,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556562 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:13:35,340::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:13:36,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:36,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923305 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:13:36,655::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:13:36,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:13:36,677::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663051 s, 560 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:13:36,679::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683366 s, 765 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:13:37,156::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:13:37,157::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:13:38,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:38,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0975567 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:40,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:40,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913274 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:13:41,433::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:13:42,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:42,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917534 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:44,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:44,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916565 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:13:46,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:13:46,693::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:13:46,704::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00157838 s, 235 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:13:46,705::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000778266 s, 672 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:46,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:47,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919459 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:13:47,168::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:13:47,169::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:13:49,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:49,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912807 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:13:50,344::task::595::Storage.TaskManager.Task::(_updateState) Task=`33b05b67-0861-4dd8-92d0-642a4bd2f1ce`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:13:50,344::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:13:50,347::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:13:50,347::task::1191::Storage.TaskManager.Task::(prepare) Task=`33b05b67-0861-4dd8-92d0-642a4bd2f1ce`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:13:50,348::task::595::Storage.TaskManager.Task::(_updateState) Task=`33b05b67-0861-4dd8-92d0-642a4bd2f1ce`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:13:50,348::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:13:50,348::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:13:50,348::task::993::Storage.TaskManager.Task::(_decref) Task=`33b05b67-0861-4dd8-92d0-642a4bd2f1ce`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:13:50,376::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:13:51,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:51,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913106 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:53,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:53,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914531 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:55,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:55,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911104 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:13:56,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:13:56,721::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:13:56,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000611835 s, 606 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:13:56,735::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706218 s, 741 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:13:57,180::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:13:57,181::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:13:57,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:57,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912757 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:13:59,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:13:59,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092655 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:01,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:01,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920123 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:03,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:04,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914229 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:14:05,388::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:14:06,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:06,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561384 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:14:06,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:14:06,749::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:14:06,764::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00199657 s, 186 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:14:06,765::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000617257 s, 847 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:14:07,188::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:14:07,189::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:14:08,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:09,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550817 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:11,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:11,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912627 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:14:11,465::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:14:13,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:13,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911761 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:15,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:15,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0989121 s, 10.4 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:14:16,770::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:14:16,780::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:14:16,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641721 s, 578 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:14:16,794::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000949366 s, 551 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:14:17,200::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:14:17,201::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:14:17,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:17,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913969 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:19,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:19,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913268 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:14:20,408::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:14:21,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:22,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913642 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:24,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:24,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132602 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:26,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:26,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918889 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:14:26,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:14:26,809::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:14:26,824::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000745978 s, 497 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:14:26,825::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631305 s, 828 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:14:27,208::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:14:27,209::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:14:28,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:28,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912149 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:30,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:30,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912209 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:32,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:33,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550298 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:35,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:14:35,422::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:14:35,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.562019 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:14:36,830::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:14:36,841::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:14:36,852::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000768159 s, 483 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:14:36,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000929902 s, 562 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:14:37,217::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:14:37,218::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:14:37,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:38,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912141 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:40,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:40,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913454 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:14:41,497::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:14:42,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:42,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912981 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:44,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:44,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912277 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:46,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:46,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0996604 s, 10.3 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:14:46,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:14:46,869::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:14:46,870::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692037 s, 536 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:14:46,882::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675432 s, 774 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:14:47,228::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:14:47,229::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:14:48,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:49,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55009 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:14:50,425::task::595::Storage.TaskManager.Task::(_updateState) Task=`23dd69a6-919f-4695-92d9-e3ff78bd66f6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:14:50,425::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:14:50,428::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:14:50,429::task::1191::Storage.TaskManager.Task::(prepare) Task=`23dd69a6-919f-4695-92d9-e3ff78bd66f6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:14:50,429::task::595::Storage.TaskManager.Task::(_updateState) Task=`23dd69a6-919f-4695-92d9-e3ff78bd66f6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:14:50,429::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:14:50,429::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:14:50,429::task::993::Storage.TaskManager.Task::(_decref) Task=`23dd69a6-919f-4695-92d9-e3ff78bd66f6`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:14:50,454::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:14:51,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:51,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912541 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:53,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:53,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913297 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:55,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:55,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913104 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:14:56,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:14:56,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:14:56,899::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000761502 s, 487 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:14:56,912::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640069 s, 817 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:14:57,240::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:14:57,241::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:14:57,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:57,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912808 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:14:59,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:14:59,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091302 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:01,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:01,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913158 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:03,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:03,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912969 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:15:05,466::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:15:05,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:06,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091307 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:15:06,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:15:06,925::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:15:06,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00530028 s, 70.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:15:06,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649086 s, 806 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:15:07,250::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:15:07,251::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:15:08,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:08,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914208 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:10,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:10,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091446 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:15:11,530::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:15:12,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:12,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919477 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:14,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:14,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913079 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:16,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:15:17,132::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:15:17,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:15:17,157::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000763073 s, 685 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:15:17,162::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620361 s, 598 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:17,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552327 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:15:17,262::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:15:17,263::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:15:19,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:19,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091268 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:15:20,478::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:15:21,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:21,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911991 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:23,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:23,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912587 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:25,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:25,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091226 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:15:27,173::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:15:27,184::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:15:27,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00540881 s, 68.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:15:27,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000856431 s, 611 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:15:27,274::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:15:27,276::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:15:27,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:28,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.505307 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:30,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:30,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913054 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:32,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:32,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915985 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:34,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:34,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912369 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:15:35,491::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:15:36,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:36,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912632 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:15:37,202::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:15:37,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:15:37,224::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000726128 s, 511 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:15:37,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680835 s, 768 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:15:37,287::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:15:37,288::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:15:38,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:38,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912483 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:40,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:40,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919143 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:15:41,567::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:15:42,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:43,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914639 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:45,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:45,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559126 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:15:47,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:15:47,242::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:15:47,253::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632142 s, 587 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:15:47,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000723572 s, 723 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:15:47,300::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:15:47,301::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:15:47,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:47,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912899 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:49,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:49,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912614 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:15:50,494::task::595::Storage.TaskManager.Task::(_updateState) Task=`f7800f83-2d7b-426c-8054-1533628eab35`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:15:50,495::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:15:50,498::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:15:50,498::task::1191::Storage.TaskManager.Task::(prepare) Task=`f7800f83-2d7b-426c-8054-1533628eab35`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:15:50,498::task::595::Storage.TaskManager.Task::(_updateState) Task=`f7800f83-2d7b-426c-8054-1533628eab35`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:15:50,498::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:15:50,498::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:15:50,498::task::993::Storage.TaskManager.Task::(_decref) Task=`f7800f83-2d7b-426c-8054-1533628eab35`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:15:50,525::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:15:52,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:52,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921236 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:54,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:54,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913601 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:15:56,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:56,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0954067 s, 10.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:15:57,258::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:15:57,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:15:57,281::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632696 s, 586 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:15:57,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006733 s, 777 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:15:57,312::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:15:57,313::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:15:58,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:15:58,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913383 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:00,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:00,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913029 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:02,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:03,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557737 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:05,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:16:05,537::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:16:05,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543014 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:16:07,287::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:16:07,299::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:16:07,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00493904 s, 75.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:16:07,312::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000733749 s, 713 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:16:07,323::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:16:07,324::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:16:07,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:07,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914928 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:09,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:09,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915161 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:16:11,599::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:16:11,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:12,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921408 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:14,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:14,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915091 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:16,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:16,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550654 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:16:17,316::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:16:17,327::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:16:17,340::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00062441 s, 594 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:16:17,342::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00079064 s, 661 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:16:17,354::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:16:17,356::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:16:18,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:18,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918512 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:16:20,549::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:16:20,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:21,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558802 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:23,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:23,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912454 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:25,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:25,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912012 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:16:27,350::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:16:27,362::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067866 s, 547 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:16:27,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:16:27,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643886 s, 812 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:16:27,384::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:16:27,388::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:16:27,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:27,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912571 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:29,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:30,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.178965 s, 5.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:32,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:32,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913412 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:34,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:34,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091299 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:16:35,560::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:16:36,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:36,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913397 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:16:37,379::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:16:37,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000785599 s, 472 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:16:37,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:16:37,413::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00074245 s, 704 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:16:37,415::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:16:37,422::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:16:38,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:38,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.505625 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:40,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:41,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550275 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:16:41,632::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:16:43,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:43,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912477 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:45,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:45,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0962361 s, 10.6 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:16:47,413::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:16:47,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00191672 s, 194 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:16:47,438::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:16:47,436::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 13:16:47,453::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000756922 s, 691 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:16:47,457::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:16:47,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:48,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543732 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:50,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:50,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108728 s, 9.4 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:16:50,563::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2746463-f51b-4ba0-893a-522ec99f355a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:16:50,563::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:16:50,635::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:16:50,635::task::1191::Storage.TaskManager.Task::(prepare) Task=`d2746463-f51b-4ba0-893a-522ec99f355a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:16:50,635::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2746463-f51b-4ba0-893a-522ec99f355a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:16:50,635::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:16:50,635::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:16:50,635::task::993::Storage.TaskManager.Task::(_decref) Task=`d2746463-f51b-4ba0-893a-522ec99f355a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:16:50,662::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:16:52,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:52,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913955 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:54,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:54,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912536 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:56,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:56,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923274 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:16:57,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:16:57,455::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000738652 s, 502 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:16:57,458::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 13:16:57,461::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >GuestMonitor-vm11::ERROR::2014-07-14 13:16:57,465::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 13:16:57,484::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:16:57,485::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:16:57,488::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:16:57,490::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:16:57,490::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:16:57,490::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:16:57,491::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:16:57,503::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006658 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:16:58,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:16:59,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914088 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:01,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:01,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914193 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:03,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:03,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914033 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:05,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:05,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913101 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:17:05,675::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:17:07,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:17:07,465::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:17:07,466::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:17:07,491::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:07,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103695 s, 9.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:17:07,508::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00285619 s, 130 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:17:07,515::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:17:07,529::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682871 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:09,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:09,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914353 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:11,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:17:11,671::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:17:12,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556743 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:14,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:14,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920218 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:16,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:16,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0985579 s, 10.4 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:17:17,478::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:17:17,479::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:17:17,516::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:17:17,528::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652338 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:17:17,535::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:17:17,547::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681917 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:18,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:18,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914222 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:20,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:17:20,688::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:17:20,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923129 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:22,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:22,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913042 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:24,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:24,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091333 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:27,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:27,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922066 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:17:27,483::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:17:27,484::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:17:27,530::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:17:27,543::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:17:27,544::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:17:27,547::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:17:27,549::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:17:27,549::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:17:27,549::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:17:27,551::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:17:27,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:17:27,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000702137 s, 528 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:17:27,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069611 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:29,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:29,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913813 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:31,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:31,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915373 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:33,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:33,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913835 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:35,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:35,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911984 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:17:35,701::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:17:37,496::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:17:37,497::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:17:37,579::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:17:37,590::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:17:37,605::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000606094 s, 612 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:37,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:17:37,619::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00550854 s, 94.9 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:37,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0947582 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:39,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:39,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914382 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:17:41,703::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:17:41,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:41,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913194 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:43,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:44,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918548 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:46,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:46,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101875 s, 10.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:17:47,503::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:17:47,504::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:17:47,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:17:47,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690796 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:17:47,711::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:17:47,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683481 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:48,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:48,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914521 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:50,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:50,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914029 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:17:50,704::task::595::Storage.TaskManager.Task::(_updateState) Task=`53715f25-2a12-4b4d-a4d9-bb2c17b77fb3`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:17:50,704::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:17:50,707::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:17:50,707::task::1191::Storage.TaskManager.Task::(prepare) Task=`53715f25-2a12-4b4d-a4d9-bb2c17b77fb3`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:17:50,707::task::595::Storage.TaskManager.Task::(_updateState) Task=`53715f25-2a12-4b4d-a4d9-bb2c17b77fb3`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:17:50,708::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:17:50,708::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:17:50,708::task::993::Storage.TaskManager.Task::(_decref) Task=`53715f25-2a12-4b4d-a4d9-bb2c17b77fb3`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:17:50,735::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:17:52,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:52,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091399 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:54,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:55,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0972948 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:57,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:57,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911925 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:17:57,516::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:17:57,517::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:17:57,636::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:17:57,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653676 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:17:57,726::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:17:57,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000729388 s, 717 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:17:59,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:17:59,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916225 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:01,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:01,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912593 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:03,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:03,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913613 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:18:05,747::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:18:05,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:05,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091393 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:18:07,529::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:18:07,530::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:18:07,652::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:18:07,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643387 s, 577 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:18:07,742::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:18:07,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000628723 s, 832 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:07,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:07,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914838 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:09,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:10,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914827 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:18:11,737::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:18:12,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:12,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913998 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:14,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:14,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912629 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:16,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:16,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0987174 s, 10.4 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:18:17,540::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:18:17,541::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:18:17,668::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:18:17,680::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668975 s, 555 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:18:17,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:18:17,769::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065976 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:18,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:18,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911141 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:20,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:18:20,760::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:18:20,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924413 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:22,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:22,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912189 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:24,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:24,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911904 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:27,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:27,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921997 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:18:27,552::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:18:27,553::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:18:27,683::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:18:27,695::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000695141 s, 534 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:18:27,775::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:18:27,787::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674245 s, 776 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:29,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:29,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913511 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:31,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:31,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913753 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:33,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:33,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915305 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:35,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:35,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911492 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:18:35,773::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:18:37,564::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:18:37,565::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:18:37,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:18:37,713::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:37,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103513 s, 9.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:18:37,733::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00232092 s, 160 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:18:37,791::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:18:37,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639044 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:39,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:40,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549619 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:18:41,771::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:18:42,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:42,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911972 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:44,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:44,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:46,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:46,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913004 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:18:47,574::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:18:47,575::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:18:47,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:18:47,750::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683938 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:18:47,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:18:47,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687635 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:48,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:48,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913086 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:18:50,777::task::595::Storage.TaskManager.Task::(_updateState) Task=`0528b5fe-9be0-4d49-b17e-3e2e4b5abeb0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:18:50,778::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:18:50,780::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:18:50,780::task::1191::Storage.TaskManager.Task::(prepare) Task=`0528b5fe-9be0-4d49-b17e-3e2e4b5abeb0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:18:50,780::task::595::Storage.TaskManager.Task::(_updateState) Task=`0528b5fe-9be0-4d49-b17e-3e2e4b5abeb0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:18:50,781::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:18:50,781::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:18:50,781::task::993::Storage.TaskManager.Task::(_decref) Task=`0528b5fe-9be0-4d49-b17e-3e2e4b5abeb0`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 13:18:50,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:18:50,820::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:18:51,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504148 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:53,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:53,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910519 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:55,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:56,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544125 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:18:57,586::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:18:57,587::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:18:57,753::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:18:57,764::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650519 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:18:57,820::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:18:57,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000715373 s, 731 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:18:58,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:18:58,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09152 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:00,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:00,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912406 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:02,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:02,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913415 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:04,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:04,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911769 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:19:05,832::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:19:06,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:06,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912485 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:19:07,594::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:19:07,595::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:19:07,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:19:07,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618027 s, 600 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:19:07,836::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:19:07,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685764 s, 763 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:08,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:08,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911863 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:10,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:10,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913061 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:19:11,803::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:19:12,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:12,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911776 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:14,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:15,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913357 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:17,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:17,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913858 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:19:17,604::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:19:17,605::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:19:17,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:19:17,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000606794 s, 611 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:19:17,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:19:17,862::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667809 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:19,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:19,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915197 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:19:20,846::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:19:21,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:21,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557782 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:23,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:24,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913015 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:26,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:26,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912619 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:19:27,614::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:19:27,615::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:19:27,798::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:19:27,810::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650305 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:19:27,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:19:27,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670031 s, 781 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:28,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:28,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912579 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:30,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:30,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912195 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:32,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:32,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912482 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:34,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:34,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091253 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:19:35,859::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:19:36,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:36,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091322 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:19:37,624::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:19:37,625::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:19:37,814::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:19:37,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678573 s, 547 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:19:37,881::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:19:37,893::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00071974 s, 727 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:38,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:38,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913197 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:40,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:19:41,835::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:19:42,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36346 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:44,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:44,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912561 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:46,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:46,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54604 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:19:47,634::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:19:47,635::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:19:47,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:19:47,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699199 s, 531 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:19:47,896::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:19:47,908::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669015 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:48,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:49,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912718 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:19:50,862::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a971b70-14a4-4c2e-ba97-5d8feb79f33e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:19:50,862::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:19:50,865::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:19:50,865::task::1191::Storage.TaskManager.Task::(prepare) Task=`7a971b70-14a4-4c2e-ba97-5d8feb79f33e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:19:50,865::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a971b70-14a4-4c2e-ba97-5d8feb79f33e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:19:50,865::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:19:50,865::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:19:50,865::task::993::Storage.TaskManager.Task::(_decref) Task=`7a971b70-14a4-4c2e-ba97-5d8feb79f33e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:19:50,892::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:19:51,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:51,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912224 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:53,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:53,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913931 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:55,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:55,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912347 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:57,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:57,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913052 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:19:57,646::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:19:57,648::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:19:57,843::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:19:57,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678441 s, 547 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:19:57,911::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:19:57,922::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645298 s, 810 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:19:59,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:19:59,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0987123 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:01,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:02,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557461 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:04,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:04,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912862 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:20:05,904::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:20:06,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:06,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912972 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:20:07,659::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:20:07,660::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:20:07,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:20:07,870::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628991 s, 590 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:20:07,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:20:07,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645529 s, 810 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:08,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:08,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091299 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:10,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:10,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913405 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:20:11,873::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:20:12,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:12,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912651 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:14,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:14,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913479 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:16,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:17,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912484 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:20:17,670::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:20:17,671::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:20:17,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:20:17,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000611365 s, 607 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:20:17,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:20:17,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658431 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:19,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:19,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925629 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:20:20,917::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:20:21,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:21,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.174218 s, 5.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:23,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:24,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36655 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:26,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:26,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921517 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:20:27,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:20:27,684::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:20:27,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:20:27,900::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663655 s, 559 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:20:27,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:20:27,968::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664597 s, 787 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:28,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:28,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920259 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:31,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:31,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914845 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:33,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:33,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542875 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:35,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:35,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923307 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:20:35,929::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:20:37,694::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:20:37,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:20:37,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:20:37,917::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:37,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103478 s, 9.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:20:37,931::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000714134 s, 520 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:20:37,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:20:37,984::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00104222 s, 502 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:39,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:40,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091215 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:20:41,917::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:20:42,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:42,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913962 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:44,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:44,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912276 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:46,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:46,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100441 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:20:47,706::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:20:47,708::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:20:47,937::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:20:47,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639117 s, 580 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:20:47,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:20:47,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669577 s, 781 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:48,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:48,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911293 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:50,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:50,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091236 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:20:50,932::task::595::Storage.TaskManager.Task::(_updateState) Task=`2fbc624c-3322-4cde-bf0f-cb2182a69262`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:20:50,932::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:20:50,935::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:20:50,935::task::1191::Storage.TaskManager.Task::(prepare) Task=`2fbc624c-3322-4cde-bf0f-cb2182a69262`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:20:50,936::task::595::Storage.TaskManager.Task::(_updateState) Task=`2fbc624c-3322-4cde-bf0f-cb2182a69262`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:20:50,936::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:20:50,936::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:20:50,936::task::993::Storage.TaskManager.Task::(_decref) Task=`2fbc624c-3322-4cde-bf0f-cb2182a69262`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:20:50,963::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:20:52,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:52,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913241 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:54,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:54,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091153 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:57,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:57,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913257 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:20:57,719::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:20:57,720::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:20:57,952::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:20:57,963::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000700573 s, 530 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:20:58,002::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:20:58,014::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000734973 s, 712 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:20:59,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:20:59,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913716 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:01,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:01,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.609926 s, 1.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:03,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:03,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091898 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:21:05,975::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:21:05,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:06,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914306 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:21:07,726::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:21:07,727::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:21:07,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:21:07,979::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644599 s, 576 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:21:08,018::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:21:08,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681893 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:08,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:08,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913018 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:10,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:10,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913569 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:21:11,951::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:21:12,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:12,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914254 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:14,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:14,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913778 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:16,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:16,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914193 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:21:17,735::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:21:17,736::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:21:17,982::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:21:17,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665875 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:21:18,034::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:21:18,045::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068878 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:18,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:18,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912587 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:20,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:20,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912262 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:21:20,987::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:21:22,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:23,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914017 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:25,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:25,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913868 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:27,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:27,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913124 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:21:27,745::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:21:27,746::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:21:27,997::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:21:28,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000715274 s, 519 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:21:28,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:21:28,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000715945 s, 731 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:29,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:29,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912141 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:31,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:31,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912548 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:33,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:33,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913531 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:35,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:35,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914069 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:21:36,000::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:21:37,757::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:21:37,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:21:37,770::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:21:37,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913425 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:21:38,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:21:38,024::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661622 s, 561 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:21:38,065::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:21:38,076::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686455 s, 762 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:39,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:39,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913896 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:21:41,983::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:21:41,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:43,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.4077 s, 727 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:45,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:45,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912609 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:47,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:47,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913104 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:21:47,766::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:21:47,767::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:21:48,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:21:48,040::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000751314 s, 494 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:21:48,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:21:48,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672077 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:49,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:49,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915816 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:21:51,003::task::595::Storage.TaskManager.Task::(_updateState) Task=`7973c6a3-57a9-4fb4-a6ba-8025395a8444`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:21:51,003::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:21:51,008::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:21:51,009::task::1191::Storage.TaskManager.Task::(prepare) Task=`7973c6a3-57a9-4fb4-a6ba-8025395a8444`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:21:51,009::task::595::Storage.TaskManager.Task::(_updateState) Task=`7973c6a3-57a9-4fb4-a6ba-8025395a8444`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:21:51,009::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:21:51,009::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:21:51,009::task::993::Storage.TaskManager.Task::(_decref) Task=`7973c6a3-57a9-4fb4-a6ba-8025395a8444`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:21:51,037::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:21:51,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:51,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912252 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:53,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:54,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552015 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:56,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:56,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912683 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:21:57,777::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:21:57,778::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:21:58,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:21:58,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632278 s, 587 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:21:58,093::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:21:58,110::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:21:58,111::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:21:58,114::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:21:58,115::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:21:58,116::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:21:58,116::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:21:58,117::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:21:58,128::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000622708 s, 840 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:21:58,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:21:58,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912319 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:00,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:01,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557803 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:03,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:03,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914813 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:05,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:05,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913195 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:22:06,049::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:22:07,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:07,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913337 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:22:07,786::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:22:07,788::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:22:08,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:22:08,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000613002 s, 605 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:22:08,133::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:22:08,145::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677003 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:09,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:09,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914204 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:11,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:11,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912513 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:22:12,016::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:22:13,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:14,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913307 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:16,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:16,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0980165 s, 10.4 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:22:17,798::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:22:17,800::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:22:18,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:22:18,086::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635589 s, 584 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:22:18,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:22:18,160::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687267 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:18,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:18,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913276 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:20,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:20,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544329 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:22:21,060::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:22:22,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:23,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915677 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:25,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:25,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54582 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:27,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:27,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914688 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:22:27,807::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:22:27,808::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:22:28,088::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:22:28,103::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:22:28,104::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:22:28,107::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:22:28,109::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:22:28,109::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:22:28,109::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:22:28,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:22:28,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000696735 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:22:28,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:22:28,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000709199 s, 737 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:29,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:29,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914116 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:31,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:32,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914819 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:34,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:34,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913861 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:22:36,077::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:22:36,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:36,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914834 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:22:37,819::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:22:37,820::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:22:38,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:22:38,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000709187 s, 523 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:22:38,180::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:22:38,193::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676038 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:38,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:38,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091291 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:40,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:40,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091388 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:22:42,054::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:22:42,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:42,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914421 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:44,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:44,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914485 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:46,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:46,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091384 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:22:47,831::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:22:47,833::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:22:48,141::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:22:48,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067413 s, 550 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:22:48,196::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:22:48,208::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690167 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:48,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:49,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546205 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:22:51,080::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf6833a1-93c3-4196-8bc7-3a3866ddec60`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:22:51,080::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:22:51,083::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:22:51,084::task::1191::Storage.TaskManager.Task::(prepare) Task=`bf6833a1-93c3-4196-8bc7-3a3866ddec60`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:22:51,084::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf6833a1-93c3-4196-8bc7-3a3866ddec60`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:22:51,084::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:22:51,084::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:22:51,084::task::993::Storage.TaskManager.Task::(_decref) Task=`bf6833a1-93c3-4196-8bc7-3a3866ddec60`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:22:51,110::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:22:51,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:51,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542314 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:54,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:54,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913762 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:56,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:56,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913844 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:22:57,844::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:22:57,845::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:22:58,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:22:58,167::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690224 s, 538 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:22:58,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:22:58,223::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665268 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:22:58,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:22:58,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911936 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:00,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:00,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091235 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:02,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:02,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913037 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:04,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:05,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.3645 s, 750 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:23:06,121::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:23:07,852::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:23:07,853::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:23:07,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:08,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548101 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:23:09,047::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:23:09,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:23:09,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697658 s, 750 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:23:09,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647655 s, 573 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:10,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:11,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548523 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:23:12,085::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:23:13,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:13,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925803 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:15,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:15,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091425 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:17,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:17,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917764 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:23:17,859::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:23:17,860::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:23:19,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:23:19,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:23:19,104::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00401138 s, 92.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:23:19,106::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672726 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:19,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:19,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912652 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:23:21,134::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:23:21,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:21,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912474 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:23,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:23,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913441 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:25,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:25,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911524 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:23:27,872::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:23:27,873::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:23:27,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:28,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091227 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:23:29,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:23:29,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:23:29,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685099 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:23:29,133::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640281 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:30,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:30,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911381 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:32,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:32,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913158 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:34,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:34,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911976 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:23:36,147::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:23:36,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:36,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914157 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:23:37,883::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:23:37,884::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:23:38,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:39,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545447 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:23:39,135::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:23:39,147::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000756384 s, 490 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:23:39,149::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:23:39,162::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658747 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:41,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:41,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0930581 s, 11.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:23:42,123::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:23:43,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:43,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912333 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:45,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:45,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911256 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:47,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:47,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912396 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:23:47,893::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:23:47,894::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 13:23:49,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:23:49,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665336 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:23:49,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:23:49,190::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643117 s, 813 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:49,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:49,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091339 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:23:51,152::task::595::Storage.TaskManager.Task::(_updateState) Task=`85b74865-02dc-4055-9f32-1daf67745e2e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:23:51,152::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:23:51,156::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:23:51,156::task::1191::Storage.TaskManager.Task::(prepare) Task=`85b74865-02dc-4055-9f32-1daf67745e2e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:23:51,156::task::595::Storage.TaskManager.Task::(_updateState) Task=`85b74865-02dc-4055-9f32-1daf67745e2e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:23:51,156::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:23:51,156::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:23:51,157::task::993::Storage.TaskManager.Task::(_decref) Task=`85b74865-02dc-4055-9f32-1daf67745e2e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:23:51,184::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:23:51,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:51,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914534 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:53,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:53,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911753 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:23:55,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:56,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912858 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:23:57,904::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:23:57,905::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:23:58,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:23:58,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917491 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:23:59,191::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:23:59,204::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000730422 s, 508 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:23:59,207::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:23:59,219::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671535 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:00,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:00,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916152 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:02,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:02,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913834 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:04,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:04,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913619 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:24:06,197::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:24:06,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:06,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914873 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:24:07,911::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:24:07,912::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:24:08,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:08,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917283 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:24:09,220::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:24:09,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665045 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:24:09,234::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:24:09,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000711816 s, 735 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:10,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:24:12,154::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:24:12,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39813 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:14,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:14,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914386 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:16,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:16,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101246 s, 10.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:24:17,923::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:24:17,924::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:24:18,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:18,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916269 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:24:19,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:24:19,260::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:24:19,271::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000634855 s, 584 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:24:19,272::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000958071 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:20,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:20,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913819 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:24:21,210::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:24:22,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:22,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911978 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:24,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:25,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916769 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:27,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:27,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914974 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:24:27,931::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:24:27,932::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:24:29,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:29,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914654 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:24:29,276::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:24:29,287::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:24:29,298::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00476366 s, 77.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:24:29,300::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000726002 s, 720 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:31,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:31,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913593 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:33,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:33,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912804 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:35,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:36,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559039 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:24:36,232::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:24:37,942::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:24:37,944::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:24:38,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:38,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913849 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:24:39,304::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:24:39,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:24:39,327::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00270346 s, 137 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:24:39,329::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000701391 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:40,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:40,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091328 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:24:42,184::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:24:42,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:42,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913341 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:44,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:44,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09139 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:46,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:47,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551841 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:24:47,955::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:24:47,956::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:24:49,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:24:49,685::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:24:49,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:24:49,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000698109 s, 531 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:24:49,718::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688702 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:49,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.59894 s, 1.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:24:51,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`a7245077-e097-44d3-b834-68da7c74cf70`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:24:51,236::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:24:51,239::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:24:51,239::task::1191::Storage.TaskManager.Task::(prepare) Task=`a7245077-e097-44d3-b834-68da7c74cf70`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:24:51,239::task::595::Storage.TaskManager.Task::(_updateState) Task=`a7245077-e097-44d3-b834-68da7c74cf70`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:24:51,239::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:24:51,239::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:24:51,240::task::993::Storage.TaskManager.Task::(_decref) Task=`a7245077-e097-44d3-b834-68da7c74cf70`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:24:51,267::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:24:51,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:51,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912819 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:53,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:53,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912494 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:24:55,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:56,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912878 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:24:57,966::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:24:57,967::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:24:58,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:24:58,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914059 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:24:59,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:24:59,734::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:24:59,745::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00610793 s, 60.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:24:59,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0010125 s, 517 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:00,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:00,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913499 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:02,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:02,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914128 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:04,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:04,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913611 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:25:06,279::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:25:06,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:06,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912731 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:25:07,978::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:25:07,980::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:25:08,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:08,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912836 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:25:09,750::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:25:09,763::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:25:09,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000746222 s, 497 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:25:09,776::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678958 s, 770 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:10,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:10,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091434 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:25:12,217::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:25:12,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:13,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54543 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:15,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:16,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549022 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:25:17,990::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:25:17,991::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:25:18,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:18,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914116 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:25:19,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:25:19,790::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:25:19,801::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000814767 s, 455 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:25:19,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00105291 s, 497 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:20,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:20,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543406 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:25:21,292::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:25:22,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:23,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546048 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:25,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:25,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914786 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:27,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:27,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911802 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:25:27,999::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:25:28,000::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:25:29,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:25:30,283::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:25:30,296::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000806601 s, 460 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:25:30,300::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:25:30,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648079 s, 807 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:30,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.615769 s, 1.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:32,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:33,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36384 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:35,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:35,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918688 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:25:36,304::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:25:37,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:25:38,012::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:25:38,013::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:25:38,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556278 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:25:40,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:25:40,327::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670756 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:25:40,330::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:25:40,342::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066499 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:40,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:40,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915559 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:25:42,249::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:25:42,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:43,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561665 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:45,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:45,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914002 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:47,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:47,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912876 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:25:48,024::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:25:48,025::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:25:49,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:49,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543001 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:25:50,343::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:25:50,355::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:25:50,370::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00073983 s, 501 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:25:50,373::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699529 s, 748 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:25:51,307::task::595::Storage.TaskManager.Task::(_updateState) Task=`2bb5cfe4-cd44-4d24-9aea-7fb89e283e31`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:25:51,307::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:25:51,310::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:25:51,310::task::1191::Storage.TaskManager.Task::(prepare) Task=`2bb5cfe4-cd44-4d24-9aea-7fb89e283e31`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:25:51,310::task::595::Storage.TaskManager.Task::(_updateState) Task=`2bb5cfe4-cd44-4d24-9aea-7fb89e283e31`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:25:51,310::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:25:51,311::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:25:51,311::task::993::Storage.TaskManager.Task::(_decref) Task=`2bb5cfe4-cd44-4d24-9aea-7fb89e283e31`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:25:51,338::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:25:51,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:52,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913503 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:54,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:54,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091391 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:25:56,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:56,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912807 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:25:58,031::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:25:58,032::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:25:58,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:25:58,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916934 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:26:00,375::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:26:00,388::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691078 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:26:00,391::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:26:00,403::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000770757 s, 679 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:00,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:00,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913377 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:02,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:02,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914143 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:04,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:04,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911778 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:26:06,350::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:26:06,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:06,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915156 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:26:08,043::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:26:08,045::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:26:08,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:08,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913759 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:26:10,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:26:10,466::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:26:10,481::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699749 s, 747 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:26:10,484::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642564 s, 577 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:11,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:11,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916973 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:26:12,281::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:26:13,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:13,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912609 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:15,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:15,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0995799 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:17,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:17,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091316 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:26:18,053::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:26:18,054::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:26:19,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:19,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911494 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:26:20,491::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:26:20,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:26:20,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000739966 s, 501 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:26:20,515::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000634976 s, 824 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:26:21,362::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:26:21,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:22,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913089 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:24,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:24,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913581 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:26,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:26,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913445 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:26:28,065::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:26:28,066::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:26:28,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:28,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918757 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:30,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:30,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913868 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:26:30,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:26:30,531::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:26:30,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000745481 s, 498 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:26:30,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698338 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:32,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:32,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091342 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:34,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:34,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912609 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:26:36,374::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:26:36,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:36,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913945 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:26:38,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:26:38,077::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:26:38,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:39,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550879 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:26:40,548::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:26:40,560::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:26:40,570::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000633766 s, 585 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:26:40,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000883203 s, 592 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:41,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:41,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912111 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:26:42,317::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:26:43,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:43,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912179 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:45,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:45,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967354 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:47,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:48,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915127 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:26:48,084::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:26:48,085::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:26:50,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:50,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911437 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:26:50,578::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:26:50,589::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:26:50,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697288 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:26:50,601::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693243 s, 754 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:26:51,380::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b31e83e-e1f7-4186-b966-afcf7992b511`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:26:51,380::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:26:51,383::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:26:51,383::task::1191::Storage.TaskManager.Task::(prepare) Task=`4b31e83e-e1f7-4186-b966-afcf7992b511`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:26:51,383::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b31e83e-e1f7-4186-b966-afcf7992b511`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:26:51,384::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:26:51,384::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:26:51,384::task::993::Storage.TaskManager.Task::(_decref) Task=`4b31e83e-e1f7-4186-b966-afcf7992b511`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:26:51,409::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:26:52,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:52,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913317 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:54,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:54,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912696 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:26:56,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:56,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913106 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:26:58,092::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:26:58,093::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:26:58,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:26:59,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557282 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:27:00,605::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 13:27:00,607::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:27:00,627::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068232 s, 544 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:27:00,640::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:27:00,641::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:27:00,644::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:27:00,646::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:27:00,646::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:27:00,646::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:27:00,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:27:00,659::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640891 s, 816 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:01,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:02,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38902 s, 737 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:04,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:04,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921337 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:27:06,421::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:27:06,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:06,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914345 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:27:08,103::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:27:08,104::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:27:08,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:08,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091291 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:27:10,636::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:27:10,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646476 s, 574 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:27:10,662::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:27:10,674::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621991 s, 841 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:10,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:11,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544538 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:27:12,350::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:27:13,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:13,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091347 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:15,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:15,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.145766 s, 7.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:17,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:27:18,114::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:27:18,116::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:27:18,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.508632 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:20,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:20,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913403 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:27:20,651::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:27:20,663::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667418 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:27:20,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:27:20,689::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690512 s, 757 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:27:21,433::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:27:22,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:22,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912348 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:24,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:24,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912728 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:26,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:26,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912188 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:27:28,127::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:27:28,128::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:27:28,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:29,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551479 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:27:30,665::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:27:30,678::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:27:30,679::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:27:30,682::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:27:30,684::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:27:30,684::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:27:30,684::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:27:30,685::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:27:30,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663866 s, 559 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:27:30,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:27:30,715::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000598795 s, 873 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:31,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:31,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912644 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:33,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:34,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542035 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:36,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:27:36,445::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:27:36,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547222 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:27:38,135::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:27:38,136::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:27:38,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:39,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545935 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:27:40,717::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:27:40,729::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:27:40,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000791046 s, 469 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:27:40,741::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000780649 s, 670 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:41,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:41,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913569 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:27:42,383::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:27:43,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:43,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913245 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:45,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:45,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0991599 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:47,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:47,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.112528 s, 9.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:27:48,148::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:27:48,149::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:27:49,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:50,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556709 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:27:50,745::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:27:50,756::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:27:50,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000592159 s, 627 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:27:50,769::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000609543 s, 858 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:27:51,448::task::595::Storage.TaskManager.Task::(_updateState) Task=`b13afeac-34fd-4b50-bacf-9e180c529faf`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:27:51,448::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:27:51,451::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:27:51,451::task::1191::Storage.TaskManager.Task::(prepare) Task=`b13afeac-34fd-4b50-bacf-9e180c529faf`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:27:51,451::task::595::Storage.TaskManager.Task::(_updateState) Task=`b13afeac-34fd-4b50-bacf-9e180c529faf`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:27:51,451::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:27:51,451::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:27:51,452::task::993::Storage.TaskManager.Task::(_decref) Task=`b13afeac-34fd-4b50-bacf-9e180c529faf`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:27:51,479::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:27:52,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:52,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912836 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:54,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:55,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560392 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:27:57,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:57,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549008 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:27:58,161::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:27:58,162::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:27:59,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:27:59,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913331 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:28:00,771::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:28:00,783::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665922 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:28:00,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:28:00,798::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000818307 s, 639 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:01,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:01,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912762 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:03,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:04,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912522 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:06,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:06,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918854 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:28:06,491::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:28:08,173::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:28:08,174::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:28:08,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:08,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913775 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:10,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:10,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912664 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:28:10,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:28:10,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:28:10,823::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00495291 s, 74.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:28:10,824::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000808986 s, 646 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:12,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:28:12,421::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:28:12,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913644 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:14,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:15,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556542 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:17,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:17,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0935303 s, 10.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:28:18,181::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:28:18,182::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:28:19,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:19,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913195 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:28:20,829::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:28:20,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:28:20,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657538 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:28:20,853::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681255 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:21,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:21,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912451 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:28:21,504::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:28:23,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:23,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912921 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:25,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:26,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504354 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:28,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:28:28,194::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:28:28,195::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:28:28,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913778 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:30,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:30,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913889 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:28:30,856::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:28:30,868::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621667 s, 597 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:28:30,871::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:28:30,884::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000763626 s, 685 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:32,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:32,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913731 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:34,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:35,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541701 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:28:36,515::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:28:37,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:37,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913761 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:28:38,201::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:28:38,202::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:28:39,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:39,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912374 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:28:40,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:28:40,897::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:28:40,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000779875 s, 476 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:28:40,912::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000701253 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:41,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:41,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912183 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:28:42,454::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:28:43,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:43,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912954 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:45,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:45,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0981263 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:47,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:47,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919672 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:28:48,211::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:28:48,212::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:28:49,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:49,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091236 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:28:50,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:28:50,924::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:28:50,935::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650711 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:28:50,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00077245 s, 677 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:28:51,518::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e7c6ac0-ce60-4467-a498-7e1bf13a19e9`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:28:51,518::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:28:51,521::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:28:51,521::task::1191::Storage.TaskManager.Task::(prepare) Task=`5e7c6ac0-ce60-4467-a498-7e1bf13a19e9`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:28:51,521::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e7c6ac0-ce60-4467-a498-7e1bf13a19e9`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:28:51,522::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:28:51,522::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:28:51,522::task::993::Storage.TaskManager.Task::(_decref) Task=`5e7c6ac0-ce60-4467-a498-7e1bf13a19e9`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:28:51,547::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:28:51,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:52,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912961 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:54,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:54,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913805 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:28:56,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:56,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912573 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:28:58,223::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:28:58,224::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:28:58,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:28:58,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547469 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:00,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:29:01,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:29:01,054::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:01,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0955089 s, 10.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:29:01,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00642944 s, 57.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:29:01,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00633657 s, 82.5 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:03,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:03,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912158 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:05,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:05,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914346 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:29:06,558::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:29:07,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:29:08,226::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:29:08,228::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:29:08,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.3981 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:10,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:10,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913091 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:29:11,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:29:11,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:29:11,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00555014 s, 66.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:29:11,178::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00117953 s, 443 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:29:12,487::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:29:12,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:13,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54639 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:15,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:15,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10024 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:17,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:17,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913259 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:29:18,238::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:29:18,239::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:29:19,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:20,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546954 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:29:21,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:29:21,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:29:21,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000714967 s, 519 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:29:21,212::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00073276 s, 714 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:29:21,570::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:29:22,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:22,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914028 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:24,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:24,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913771 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:26,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:26,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912577 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:29:28,250::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:29:28,251::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:29:28,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:28,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912416 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:30,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:31,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914427 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:29:31,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:29:31,227::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:29:31,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00541342 s, 68.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:29:31,241::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000930948 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:33,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:33,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542894 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:35,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:35,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912471 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:29:36,581::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:29:37,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:37,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911666 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:29:38,261::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:29:38,262::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:29:39,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:39,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091306 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:29:41,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:29:41,257::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:29:41,258::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654861 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:29:41,271::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00100809 s, 519 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:41,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:42,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917827 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:29:42,519::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:29:44,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:44,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914172 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:46,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:46,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0996689 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:29:48,273::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:29:48,274::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:29:48,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:48,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911661 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:50,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:50,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912898 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:29:51,273::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:29:51,284::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:29:51,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674446 s, 550 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:29:51,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674846 s, 775 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:29:51,584::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5ca3c32-8ee4-49a0-a82d-ab2c4d3916c5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:29:51,585::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:29:51,587::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:29:51,587::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5ca3c32-8ee4-49a0-a82d-ab2c4d3916c5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:29:51,587::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5ca3c32-8ee4-49a0-a82d-ab2c4d3916c5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:29:51,587::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:29:51,588::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:29:51,588::task::993::Storage.TaskManager.Task::(_decref) Task=`a5ca3c32-8ee4-49a0-a82d-ab2c4d3916c5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:29:51,613::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:29:52,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:52,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923272 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:54,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:54,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912905 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:29:56,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:57,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912311 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:29:58,282::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:29:58,284::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:29:59,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:29:59,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912358 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:01,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:01,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916324 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:30:01,299::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:30:01,311::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:30:01,322::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645863 s, 574 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:30:01,324::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068654 s, 762 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:03,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:03,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913611 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:05,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:05,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914483 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:30:06,624::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:30:07,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:07,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911772 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:30:08,295::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:30:08,296::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:30:09,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:09,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092303 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:30:11,329::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:30:11,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:30:11,356::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669231 s, 554 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:30:11,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000718016 s, 728 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:11,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:11,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912791 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:30:12,552::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:30:13,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:13,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913271 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:15,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:16,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.147999 s, 6.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:18,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:30:18,305::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:30:18,306::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:30:18,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543209 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:20,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:30:21,488::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:30:21,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:30:21,503::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626692 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:30:21,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000702089 s, 745 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:21,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.587913 s, 1.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:30:21,637::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:30:23,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:23,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091234 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:25,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:25,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091343 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:27,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:27,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912443 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:30:28,314::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:30:28,315::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:30:29,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:30,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914172 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:30:31,524::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:30:31,536::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629227 s, 590 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:30:31,537::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:30:31,549::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650431 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:32,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:32,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919647 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:34,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:34,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911594 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:36,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:36,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912874 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:30:36,649::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:30:38,324::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:30:38,325::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:30:38,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:38,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913413 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:40,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:30:41,927::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:30:41,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:30:41,951::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066866 s, 555 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:30:41,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691039 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:41,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.46962 s, 697 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:30:42,583::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:30:43,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:44,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912923 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:46,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:46,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0994918 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:30:48,335::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:30:48,336::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:30:48,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:48,471::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911217 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:50,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:50,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912788 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:30:51,652::task::595::Storage.TaskManager.Task::(_updateState) Task=`28c53bc4-70af-4e47-9a52-d21ad583a8e2`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:30:51,653::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:30:51,655::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:30:51,656::task::1191::Storage.TaskManager.Task::(prepare) Task=`28c53bc4-70af-4e47-9a52-d21ad583a8e2`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:30:51,656::task::595::Storage.TaskManager.Task::(_updateState) Task=`28c53bc4-70af-4e47-9a52-d21ad583a8e2`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:30:51,656::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:30:51,656::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:30:51,656::task::993::Storage.TaskManager.Task::(_decref) Task=`28c53bc4-70af-4e47-9a52-d21ad583a8e2`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:30:51,682::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:30:51,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:30:51,974::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:30:51,974::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000741842 s, 500 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:30:51,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000737 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:52,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:52,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912569 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:54,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:54,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913317 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:30:56,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:56,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911866 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:30:58,344::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:30:58,345::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:30:58,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:30:59,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918534 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:01,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:01,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913816 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:31:01,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:31:02,000::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:31:02,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000775277 s, 479 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:31:02,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000783843 s, 667 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:03,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:03,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913473 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:05,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:05,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.495912 s, 2.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:31:06,694::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:31:07,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:31:08,356::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:31:08,357::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:31:08,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54299 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:10,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:10,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559295 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:31:12,015::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:31:12,027::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:31:12,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00527601 s, 70.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:31:12,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647905 s, 807 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:31:12,615::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:31:12,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:13,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911702 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:15,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:15,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912308 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:17,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:17,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912515 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:31:18,368::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:31:18,370::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:31:19,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:19,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912894 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:21,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:21,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913658 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:31:21,706::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:31:22,045::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:31:22,056::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:31:22,067::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00191896 s, 193 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:31:22,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069253 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:23,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:23,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913237 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:25,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:25,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912511 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:27,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:27,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913978 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:31:28,381::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:31:28,382::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:31:29,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:30,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914089 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:32,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:31:32,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:31:32,151::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:31:32,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655748 s, 798 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:31:32,169::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006081 s, 610 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:32,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.126137 s, 8.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:34,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:34,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914036 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:36,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:31:36,718::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:31:36,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50709 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:31:38,393::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:31:38,394::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:31:38,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:39,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561954 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:41,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:41,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913316 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:31:42,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:31:42,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:31:42,197::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643637 s, 813 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:31:42,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674092 s, 550 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:31:42,647::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:31:43,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:43,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914741 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:45,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:46,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0974258 s, 10.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:31:48,405::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:31:48,406::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:31:48,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:48,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914664 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:50,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:50,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913058 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:31:51,722::task::595::Storage.TaskManager.Task::(_updateState) Task=`e93f5db5-7808-4511-a5f9-86c8aad82e7c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:31:51,722::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:31:51,725::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:31:51,725::task::1191::Storage.TaskManager.Task::(prepare) Task=`e93f5db5-7808-4511-a5f9-86c8aad82e7c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:31:51,725::task::595::Storage.TaskManager.Task::(_updateState) Task=`e93f5db5-7808-4511-a5f9-86c8aad82e7c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:31:51,725::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:31:51,725::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:31:51,726::task::993::Storage.TaskManager.Task::(_decref) Task=`e93f5db5-7808-4511-a5f9-86c8aad82e7c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:31:51,751::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:31:52,205::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:31:52,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:31:52,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625403 s, 593 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:31:52,234::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00147238 s, 355 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:52,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:52,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912156 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:54,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:54,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912884 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:31:56,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:57,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913699 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:31:58,417::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:31:58,418::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:31:59,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:31:59,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913766 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:01,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:01,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55638 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:32:02,238::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 13:32:02,590::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:32:02,607::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671691 s, 552 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:32:02,613::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:32:02,614::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:32:02,617::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:32:02,619::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:32:02,619::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:32:02,619::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:32:02,620::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:32:02,632::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659411 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:03,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:03,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912094 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:05,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:05,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915103 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:32:06,764::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:32:07,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:08,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091384 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:32:08,426::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:32:08,428::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:32:10,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:10,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914887 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:12,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:12,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913609 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:32:12,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:32:12,628::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000963721 s, 385 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:32:12,635::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:32:12,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687015 s, 761 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:32:12,681::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:32:14,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:14,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922061 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:16,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:16,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912707 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:32:18,439::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:32:18,440::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:32:18,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:18,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913053 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:20,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:20,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913109 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:32:21,775::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:32:22,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:32:22,643::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648711 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:32:22,650::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:32:22,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068029 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:22,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:22,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914312 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:24,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:25,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924485 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:27,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:27,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913669 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:32:28,446::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:32:28,447::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:32:29,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:29,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912703 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:31,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:31,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913552 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:32:32,645::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:32:32,657::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:32:32,658::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:32:32,661::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:32:32,663::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:32:32,663::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:32:32,663::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:32:32,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:32:32,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:32:32,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00112106 s, 331 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:32:32,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000592866 s, 882 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:33,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:33,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.135246 s, 7.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:35,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:36,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559658 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:32:36,787::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:32:38,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:38,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913422 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:32:38,456::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:32:38,458::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:32:40,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:40,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912977 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:42,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:42,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914265 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:32:42,692::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:32:42,704::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:32:42,715::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:32:42,716::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0041734 s, 88.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:32:42,717::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000986578 s, 530 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:44,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:44,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914129 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:46,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:46,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912222 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:32:48,469::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:32:48,470::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:32:48,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:49,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556845 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:51,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:51,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912195 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:32:51,794::task::595::Storage.TaskManager.Task::(_updateState) Task=`537b8aa8-5329-4229-8441-81f3e918bd09`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:32:51,794::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:32:51,797::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:32:51,797::task::1191::Storage.TaskManager.Task::(prepare) Task=`537b8aa8-5329-4229-8441-81f3e918bd09`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:32:51,798::task::595::Storage.TaskManager.Task::(_updateState) Task=`537b8aa8-5329-4229-8441-81f3e918bd09`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:32:51,798::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:32:51,798::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:32:51,798::task::993::Storage.TaskManager.Task::(_decref) Task=`537b8aa8-5329-4229-8441-81f3e918bd09`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:32:51,824::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:32:52,721::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:32:52,732::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:32:52,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00357944 s, 104 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:32:52,744::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00102874 s, 508 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:53,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:53,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091237 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:55,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:55,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912779 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:32:57,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:32:58,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55294 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:32:58,481::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:32:58,482::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:33:00,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:00,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911934 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:02,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:02,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916623 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:33:02,748::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:33:02,761::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:33:02,761::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697893 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:33:02,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0010069 s, 519 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:04,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:04,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.214387 s, 4.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:06,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:33:06,838::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:33:07,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544324 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:33:08,493::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:33:08,494::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:33:09,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:09,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914285 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:11,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:11,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913685 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:33:12,754::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:33:12,776::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:33:12,787::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:33:12,798::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00071359 s, 520 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:33:12,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000865712 s, 604 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:13,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:14,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543853 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:16,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:16,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0980021 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:18,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:33:18,506::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:33:18,507::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:33:18,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550681 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:20,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:21,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913369 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:33:21,850::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:33:22,804::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:33:22,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:33:22,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000716146 s, 518 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:33:22,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632978 s, 826 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:23,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:23,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913278 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:25,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:25,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091435 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:27,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:27,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913516 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:33:28,514::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:33:28,515::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:33:29,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:29,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914194 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:31,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:32,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557153 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:33:32,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:33:32,844::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:33:32,855::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00600863 s, 61.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:33:32,857::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066435 s, 787 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:34,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:34,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914713 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:36,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:36,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545562 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:33:36,862::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:33:38,526::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:33:38,527::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:33:38,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:38,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912841 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:40,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:40,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913822 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:33:42,793::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:33:42,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:33:42,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:33:42,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000709536 s, 523 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:33:42,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681331 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:43,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:43,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913648 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:45,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:45,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914842 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:47,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:33:48,539::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:33:48,540::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:33:48,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36366 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:50,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:51,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550694 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:33:51,866::task::595::Storage.TaskManager.Task::(_updateState) Task=`efe4b0f6-8e79-4122-ab04-e65913e58bb5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:33:51,866::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:33:51,869::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:33:51,869::task::1191::Storage.TaskManager.Task::(prepare) Task=`efe4b0f6-8e79-4122-ab04-e65913e58bb5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:33:51,869::task::595::Storage.TaskManager.Task::(_updateState) Task=`efe4b0f6-8e79-4122-ab04-e65913e58bb5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:33:51,869::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:33:51,869::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:33:51,870::task::993::Storage.TaskManager.Task::(_decref) Task=`efe4b0f6-8e79-4122-ab04-e65913e58bb5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:33:51,895::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:33:52,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:33:52,906::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:33:52,917::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00521783 s, 71.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:33:52,919::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00103637 s, 505 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:53,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:53,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913581 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:55,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:55,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911412 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:33:57,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:33:57,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913452 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:33:58,551::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:33:58,552::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:33:59,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:00,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913028 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:02,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:02,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913852 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:34:02,923::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:34:02,935::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:34:02,946::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667712 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:34:02,947::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000748715 s, 699 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:04,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:04,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911914 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:06,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:06,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913642 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:34:06,907::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:34:08,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:08,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911502 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:34:08,569::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:34:08,574::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:34:10,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:10,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913761 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:12,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:12,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911247 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:34:12,825::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 13:34:12,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:34:12,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:34:12,980::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000616205 s, 849 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:34:12,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00071922 s, 516 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:14,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:14,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912787 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:16,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:17,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912821 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:34:18,571::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:34:18,573::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:34:19,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:19,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557658 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:21,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:34:21,919::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:34:22,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560494 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:34:22,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:34:22,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:34:23,010::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000773576 s, 676 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:34:23,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000792213 s, 468 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:24,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:24,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913691 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:26,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:26,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912197 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:28,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:28,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912851 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:34:28,583::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:34:28,584::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:34:30,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:30,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912265 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:32,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:32,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914533 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:34:33,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:34:33,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:34:33,040::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00527064 s, 99.2 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:34:33,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660272 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:34,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:34,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912213 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:36,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:34:36,942::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:34:37,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091643 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:34:38,594::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:34:38,595::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:34:39,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:39,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913317 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:41,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:41,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912545 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:34:42,858::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 13:34:43,048::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:34:43,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:34:43,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00168999 s, 309 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:34:43,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679382 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:43,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:43,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091217 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:45,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:45,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913314 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:47,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:47,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913352 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:34:48,603::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:34:48,604::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:34:49,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:49,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913655 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:51,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:51,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914496 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:34:51,946::task::595::Storage.TaskManager.Task::(_updateState) Task=`591c07f7-80eb-408d-af62-71841668fb71`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:34:51,946::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:34:51,949::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:34:51,949::task::1191::Storage.TaskManager.Task::(prepare) Task=`591c07f7-80eb-408d-af62-71841668fb71`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:34:51,949::task::595::Storage.TaskManager.Task::(_updateState) Task=`591c07f7-80eb-408d-af62-71841668fb71`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:34:51,950::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:34:51,950::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:34:51,950::task::993::Storage.TaskManager.Task::(_decref) Task=`591c07f7-80eb-408d-af62-71841668fb71`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:34:51,976::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:34:53,081::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:34:53,082::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:34:53,109::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626362 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:34:53,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00129249 s, 405 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:53,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:53,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911869 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:55,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:56,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913576 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:34:58,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:34:58,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0936767 s, 10.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:34:58,610::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:34:58,611::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:35:00,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:00,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913385 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:02,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:02,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913451 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:35:03,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:35:03,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:35:03,137::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00538477 s, 68.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:35:03,139::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000723283 s, 723 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:04,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:04,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912593 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:06,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:06,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912776 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:35:06,987::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:35:08,622::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:35:08,623::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:35:08,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:09,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542693 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:11,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:11,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091388 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:35:12,891::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:35:13,141::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:35:13,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:35:13,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704885 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:35:13,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694864 s, 753 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:13,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:13,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912692 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:16,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:16,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0962575 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:18,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:35:18,631::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:35:18,632::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:35:18,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556768 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:20,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:21,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541929 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:35:22,000::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:35:23,169::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:35:23,180::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:35:23,192::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630187 s, 589 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:35:23,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000705256 s, 742 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:23,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:23,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.532653 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:25,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:26,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091261 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:28,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:28,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913474 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:35:28,643::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:35:28,644::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:35:30,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:30,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913656 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:32,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:32,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914691 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:35:33,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:35:33,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:35:33,212::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000784396 s, 473 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:35:33,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680663 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:34,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:34,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912755 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:36,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:36,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914279 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:35:37,011::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:35:38,655::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:35:38,656::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:35:38,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:38,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918443 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:40,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:41,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558091 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:35:42,925::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:35:43,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:35:43,241::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:35:43,252::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000696558 s, 533 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:35:43,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065503 s, 798 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:43,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:43,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913002 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:45,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:45,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0963606 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:47,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:47,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917498 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:35:48,667::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:35:48,668::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:35:49,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:49,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913566 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:51,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:35:52,015::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a7ffbbe-f51a-4787-b1c1-48b2a8e723a0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:35:52,015::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:35:52,098::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:35:52,099::task::1191::Storage.TaskManager.Task::(prepare) Task=`0a7ffbbe-f51a-4787-b1c1-48b2a8e723a0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:35:52,100::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a7ffbbe-f51a-4787-b1c1-48b2a8e723a0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:35:52,101::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:35:52,101::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:35:52,102::task::993::Storage.TaskManager.Task::(_decref) Task=`0a7ffbbe-f51a-4787-b1c1-48b2a8e723a0`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 13:35:52,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967459 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:35:52,133::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:35:53,258::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:35:53,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:35:53,281::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00595758 s, 62.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:35:53,283::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662593 s, 789 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:54,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:54,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913745 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:35:56,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:56,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555325 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:35:58,675::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:35:58,676::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:35:58,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:35:58,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912241 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:00,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:01,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912058 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:03,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:03,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914702 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:36:03,285::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:36:03,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:36:03,308::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000596527 s, 622 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:36:03,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668311 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:05,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:05,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914102 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:36:07,145::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:36:07,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:07,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914203 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:36:08,687::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:36:08,688::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:36:09,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:10,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38015 s, 742 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:12,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:12,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0931851 s, 11.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:36:12,961::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:36:13,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:36:13,325::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:36:13,335::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623201 s, 595 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:36:13,338::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000737683 s, 709 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:14,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:15,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913363 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:17,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:17,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917569 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:36:18,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:36:18,696::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:36:19,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:19,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545118 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:21,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:21,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912063 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:36:22,159::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:36:23,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:36:23,352::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:36:23,363::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00147766 s, 251 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:36:23,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659571 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:23,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:23,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09128 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:25,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:26,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911596 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:28,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:28,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913125 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:36:28,704::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:36:28,705::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:36:30,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:30,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911532 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:32,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:32,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913167 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:36:33,371::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:36:33,382::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:36:33,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000710638 s, 522 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:36:33,394::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00063391 s, 825 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:34,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:34,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913058 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:36,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:36,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914615 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:36:37,171::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:36:38,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:36:38,716::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:36:38,717::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:36:38,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914119 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:40,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:40,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913256 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:42,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:36:43,001::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:36:43,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091323 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:36:43,398::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:36:43,409::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:36:43,421::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000709006 s, 523 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:36:43,422::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668841 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:45,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:45,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911413 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:47,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:47,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091322 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:36:48,725::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:36:48,726::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:36:49,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:49,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913046 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:51,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:51,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912206 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:36:52,174::task::595::Storage.TaskManager.Task::(_updateState) Task=`2a2d1583-89f4-4504-9124-867fe7b7b77f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:36:52,175::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:36:52,177::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:36:52,178::task::1191::Storage.TaskManager.Task::(prepare) Task=`2a2d1583-89f4-4504-9124-867fe7b7b77f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:36:52,178::task::595::Storage.TaskManager.Task::(_updateState) Task=`2a2d1583-89f4-4504-9124-867fe7b7b77f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:36:52,178::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:36:52,178::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:36:52,178::task::993::Storage.TaskManager.Task::(_decref) Task=`2a2d1583-89f4-4504-9124-867fe7b7b77f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:36:52,204::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:36:53,425::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:36:53,436::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:36:53,447::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000696876 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:36:53,449::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650077 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:53,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:53,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915446 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:36:55,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:56,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36306 s, 751 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:36:58,734::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:36:58,735::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:36:59,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:36:59,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910472 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:01,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:01,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546589 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:37:03,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:37:03,461::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 13:37:03,465::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000728992 s, 509 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:37:03,481::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:37:03,482::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:37:03,485::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:37:03,486::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:37:03,486::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:37:03,487::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:37:03,488::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:37:03,499::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065964 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:03,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:03,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912838 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:05,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:05,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912405 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:37:07,217::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:37:07,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:08,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912725 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:37:08,745::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:37:08,746::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:37:10,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:10,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914814 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:12,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:12,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542002 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:37:13,031::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:37:13,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:37:13,486::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670546 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:37:13,503::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:37:13,515::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681246 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:14,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:15,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544137 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:17,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:17,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912321 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:37:18,756::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:37:18,757::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:37:19,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:19,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09136 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:21,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:21,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913833 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:37:22,228::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:37:23,489::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:37:23,501::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661276 s, 561 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:37:23,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:37:23,530::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695545 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:23,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:24,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559439 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:26,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:26,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561663 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:37:28,765::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:37:28,766::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:37:28,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:28,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912558 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:30,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:31,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913752 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:33,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:33,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913121 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:37:33,504::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:37:33,537::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:37:33,539::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:37:33,540::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:37:33,557::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-29::DEBUG::2014-07-14 13:37:33,560::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000623114 s, 839 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:37:33,564::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:37:33,565::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:37:33,565::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:37:33,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:37:33,578::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651162 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:35,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:35,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912526 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:37:37,240::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:37:37,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:37,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913326 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:37:38,783::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:37:38,789::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:37:39,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:39,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912569 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:41,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:41,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913596 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:37:43,063::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 13:37:43,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:37:43,578::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663333 s, 788 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:37:43,583::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:37:43,594::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672784 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:43,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:43,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914097 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:45,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:45,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0972635 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:48,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:48,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911742 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:37:48,790::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:37:48,791::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:37:50,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:50,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091373 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:37:52,243::task::595::Storage.TaskManager.Task::(_updateState) Task=`98378b7d-6b00-463f-8f60-cc9e05d59003`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:37:52,243::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:37:52,246::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:37:52,246::task::1191::Storage.TaskManager.Task::(prepare) Task=`98378b7d-6b00-463f-8f60-cc9e05d59003`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:37:52,247::task::595::Storage.TaskManager.Task::(_updateState) Task=`98378b7d-6b00-463f-8f60-cc9e05d59003`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:37:52,247::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:37:52,247::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:37:52,247::task::993::Storage.TaskManager.Task::(_decref) Task=`98378b7d-6b00-463f-8f60-cc9e05d59003`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:37:52,274::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:37:52,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:52,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912416 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:37:53,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:37:53,596::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696846 s, 751 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:37:53,597::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:37:53,612::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000715966 s, 518 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:54,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:54,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912752 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:37:56,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:57,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557637 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:37:58,801::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:37:58,802::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:37:59,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:37:59,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913409 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:01,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:01,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910224 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:03,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:03,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913554 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:38:03,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:38:03,627::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:38:03,638::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00548412 s, 95.4 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:38:03,640::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000828773 s, 448 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:05,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:05,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913343 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:38:07,286::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:38:07,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:07,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912873 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:38:08,819::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:38:08,820::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:38:09,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:09,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913321 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:11,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:12,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559065 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:38:13,095::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 13:38:13,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:38:13,657::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:38:13,673::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000609399 s, 858 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:38:13,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680756 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:14,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:14,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912829 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:16,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:16,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915038 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:18,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:18,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091274 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:38:18,828::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:38:18,829::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:38:20,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:20,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912995 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:38:22,299::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:38:22,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:23,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912885 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:38:23,683::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:38:23,684::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:38:23,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669623 s, 781 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:38:23,710::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667556 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:25,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:25,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091213 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:27,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:27,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913467 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:38:28,840::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:38:28,841::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:38:29,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:29,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915008 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:31,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:31,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927224 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:33,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:33,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912829 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:38:33,717::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:38:33,718::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:38:33,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699027 s, 531 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:38:33,742::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632344 s, 827 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:35,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:36,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550437 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:38:37,312::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:38:38,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:38,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091222 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:38:38,852::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:38:38,853::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:38:40,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:40,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912704 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:42,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:43,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915691 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:38:43,128::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:38:43,746::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:38:43,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:38:43,759::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656136 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:38:43,772::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663981 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:45,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:45,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915623 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:47,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:47,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916043 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:38:48,860::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:38:48,861::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:38:49,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:49,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912723 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:51,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:51,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550019 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:38:52,315::task::595::Storage.TaskManager.Task::(_updateState) Task=`9ee1a92e-25f8-4dfc-b5d4-e023713a6fa0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:38:52,316::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:38:52,319::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:38:52,319::task::1191::Storage.TaskManager.Task::(prepare) Task=`9ee1a92e-25f8-4dfc-b5d4-e023713a6fa0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:38:52,319::task::595::Storage.TaskManager.Task::(_updateState) Task=`9ee1a92e-25f8-4dfc-b5d4-e023713a6fa0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:38:52,319::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:38:52,319::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:38:52,319::task::993::Storage.TaskManager.Task::(_decref) Task=`9ee1a92e-25f8-4dfc-b5d4-e023713a6fa0`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:38:52,345::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:38:53,776::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:38:53,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:38:53,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00513864 s, 72.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:38:53,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000804547 s, 650 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:53,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:54,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914348 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:56,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:56,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091315 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:38:58,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:38:58,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554647 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:38:58,869::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:38:58,870::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:39:00,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:00,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911003 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:02,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:03,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091409 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:39:04,015::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:39:04,015::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:39:04,041::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666218 s, 785 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:39:04,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652954 s, 568 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:05,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:05,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913127 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:07,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:39:07,357::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:39:07,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543785 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:39:08,877::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:39:08,878::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:39:09,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:10,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552335 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:12,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:12,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091206 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:39:13,162::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 13:39:14,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:39:14,060::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:39:14,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626546 s, 835 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:39:14,072::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622268 s, 596 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:14,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:14,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912627 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:16,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:17,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545487 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:39:18,888::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:39:18,889::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:39:19,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:19,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548827 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:21,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:22,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55183 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:39:22,370::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 13:39:24,081::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:39:24,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:39:24,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00333496 s, 157 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:39:24,103::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000624858 s, 594 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:24,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:24,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547697 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:26,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:27,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558605 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:39:28,900::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:39:28,901::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:39:29,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:29,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914398 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:31,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:31,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912094 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:33,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:33,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913956 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:39:34,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:39:34,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:39:34,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000820926 s, 637 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:39:34,132::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000922587 s, 402 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:35,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:35,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912921 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:39:37,384::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:39:37,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:37,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913172 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:39:38,913::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:39:38,914::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:39:40,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:40,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918832 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:42,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:42,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550613 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:39:43,195::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 13:39:44,137::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:39:44,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:39:44,158::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648831 s, 806 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:39:44,159::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000716712 s, 518 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:44,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:45,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545323 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:47,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:47,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916404 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:39:48,925::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:39:48,926::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:39:49,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:49,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912708 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:51,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:51,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913706 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:39:52,387::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4be7f4c-9acc-452c-bbaf-44bd279ff3e0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:39:52,387::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:39:52,390::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:39:52,390::task::1191::Storage.TaskManager.Task::(prepare) Task=`e4be7f4c-9acc-452c-bbaf-44bd279ff3e0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:39:52,390::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4be7f4c-9acc-452c-bbaf-44bd279ff3e0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:39:52,391::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:39:52,391::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:39:52,391::task::993::Storage.TaskManager.Task::(_decref) Task=`e4be7f4c-9acc-452c-bbaf-44bd279ff3e0`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:39:52,418::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:39:53,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:39:54,184::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:39:54,197::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:39:54,208::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704741 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:39:54,209::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000623357 s, 839 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:54,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.578612 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:56,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:56,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918767 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:39:58,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:39:58,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913314 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:39:58,937::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:39:58,938::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:40:00,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:01,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502288 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:03,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:03,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0934745 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:40:04,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:40:04,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00072209 s, 514 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:40:04,237::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:40:04,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637829 s, 820 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:05,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:05,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.532565 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:40:07,430::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:40:07,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:07,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912616 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:40:08,948::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:40:08,950::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:40:09,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:09,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914965 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:11,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:12,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913558 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:40:13,241::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:40:14,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:14,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914062 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:40:14,253::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:40:14,265::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:40:14,280::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629995 s, 589 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:40:14,281::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00100625 s, 520 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:16,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:16,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0973267 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:18,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:18,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912227 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:40:18,960::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:40:18,961::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:40:20,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:20,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913866 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:40:22,442::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:40:22,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:22,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913085 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:40:24,285::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:40:24,296::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:40:24,307::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705577 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:40:24,308::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640972 s, 816 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:24,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:24,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912711 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:26,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:26,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911297 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:40:28,972::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:40:28,973::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:40:29,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:29,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913875 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:31,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:31,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914573 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:33,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:33,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544381 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:40:34,311::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:40:34,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:40:34,338::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000569841 s, 651 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:40:34,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641754 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:35,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:35,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913771 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:40:37,453::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:40:37,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:38,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912975 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:40:38,985::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:40:38,986::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:40:40,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:40,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913369 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:42,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:42,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912873 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:40:43,274::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:40:44,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:40:44,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:40:44,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:40:44,410::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00073208 s, 714 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:40:44,414::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000898374 s, 413 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:44,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129923 s, 7.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:46,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:46,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910053 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:48,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:48,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912746 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:40:48,997::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:40:48,998::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:40:50,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:50,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915453 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:40:52,457::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f2eba1a-1ead-45e3-998f-adf61bd5c9fa`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:40:52,458::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:40:52,460::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:40:52,461::task::1191::Storage.TaskManager.Task::(prepare) Task=`2f2eba1a-1ead-45e3-998f-adf61bd5c9fa`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:40:52,461::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f2eba1a-1ead-45e3-998f-adf61bd5c9fa`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:40:52,461::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:40:52,461::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:40:52,461::task::993::Storage.TaskManager.Task::(_decref) Task=`2f2eba1a-1ead-45e3-998f-adf61bd5c9fa`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:40:52,487::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:40:52,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:52,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912754 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:40:54,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:40:54,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:40:54,445::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000600743 s, 618 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:40:54,448::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647781 s, 807 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:54,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:55,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912096 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:40:57,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:57,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912057 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:40:59,009::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:40:59,010::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:40:59,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:40:59,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910071 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:01,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:01,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911904 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:03,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:03,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913244 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:41:04,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:41:04,462::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:41:04,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00537662 s, 69.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:41:04,474::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647507 s, 808 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:05,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:05,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910525 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:41:07,500::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:41:07,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:07,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911757 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:41:09,022::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:41:09,023::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:41:09,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:09,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912507 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:11,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:11,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912084 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:41:13,307::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:41:13,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:14,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912411 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:41:14,479::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:41:14,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:41:14,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00372895 s, 99.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:41:14,503::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000784466 s, 667 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:16,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:16,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917423 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:18,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:18,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914022 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:41:19,029::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:41:19,030::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:41:20,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:20,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911685 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:41:22,514::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:41:22,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:22,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924597 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:41:24,508::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:41:24,520::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:41:24,536::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691033 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:41:24,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639905 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:24,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:26,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36347 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:28,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:28,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914049 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:41:29,039::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:41:29,040::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:41:30,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:30,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925971 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:32,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:32,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912623 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:34,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:34,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919164 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:41:34,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:41:34,556::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697475 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:41:34,556::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:41:34,570::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000717782 s, 729 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:36,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:36,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913273 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:41:37,532::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:41:38,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:38,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912857 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:41:39,047::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:41:39,048::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:41:40,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:40,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924347 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:42,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:42,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912772 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:41:43,341::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:41:44,572::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:41:44,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:41:44,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000771572 s, 481 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:41:44,599::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675481 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:45,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:45,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091635 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:47,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:47,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912418 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:41:49,059::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:41:49,060::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:41:49,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:49,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925347 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:51,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:51,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911754 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:41:52,535::task::595::Storage.TaskManager.Task::(_updateState) Task=`205675a8-85c5-4e95-b827-065f1ac45022`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:41:52,535::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:41:52,538::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:41:52,538::task::1191::Storage.TaskManager.Task::(prepare) Task=`205675a8-85c5-4e95-b827-065f1ac45022`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:41:52,538::task::595::Storage.TaskManager.Task::(_updateState) Task=`205675a8-85c5-4e95-b827-065f1ac45022`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:41:52,538::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:41:52,538::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:41:52,539::task::993::Storage.TaskManager.Task::(_decref) Task=`205675a8-85c5-4e95-b827-065f1ac45022`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:41:52,565::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:41:53,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:53,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914119 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:41:54,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:41:54,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:41:54,616::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688362 s, 539 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:41:54,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00072626 s, 720 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:55,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:55,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911453 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:41:57,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:41:58,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.535362 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:41:59,070::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:41:59,071::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:42:00,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:00,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912584 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:02,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:02,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913217 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:04,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:04,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091596 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:42:04,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:42:04,644::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 13:42:04,652::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000960615 s, 386 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:42:04,665::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:42:04,666::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:42:04,669::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:42:04,670::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:42:04,671::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:42:04,671::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:42:04,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:42:04,683::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642199 s, 814 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:06,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:06,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920862 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:42:07,577::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:42:08,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:42:09,080::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:42:09,081::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:42:09,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542858 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:11,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:11,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913094 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:42:13,374::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:42:13,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:13,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544781 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:42:14,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:42:14,673::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662085 s, 560 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:42:14,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:42:14,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663379 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:16,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:16,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0980053 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:18,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:18,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921621 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:42:19,089::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:42:19,090::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:42:20,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:20,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54415 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:42:22,590::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:42:22,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:23,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914464 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:42:24,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:42:24,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651183 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:42:24,702::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:42:24,713::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657135 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:25,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:25,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091041 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:27,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:27,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912878 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:42:29,100::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:42:29,101::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:42:29,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:29,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914198 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:31,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:31,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914359 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:33,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:33,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913139 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:42:34,690::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:42:34,702::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:42:34,704::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:42:34,707::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:42:34,708::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:42:34,709::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:42:34,709::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:42:34,710::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:42:34,723::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651531 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:42:34,724::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:42:34,737::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669261 s, 781 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:35,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:35,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913791 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:42:37,603::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:42:37,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:37,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918892 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:42:39,112::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:42:39,113::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:42:39,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:40,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54576 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:42,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:42,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913954 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:42:43,413::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:42:44,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:44,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913602 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:42:44,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:42:44,750::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000766979 s, 484 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:42:44,753::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:42:44,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000734098 s, 712 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:46,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:47,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556825 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:42:49,120::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:42:49,122::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:42:49,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:49,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913402 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:51,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:51,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913901 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:42:52,606::task::595::Storage.TaskManager.Task::(_updateState) Task=`443fa593-e12b-4d74-bdcf-7af1ddf47a8d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:42:52,606::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:42:52,609::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:42:52,609::task::1191::Storage.TaskManager.Task::(prepare) Task=`443fa593-e12b-4d74-bdcf-7af1ddf47a8d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:42:52,609::task::595::Storage.TaskManager.Task::(_updateState) Task=`443fa593-e12b-4d74-bdcf-7af1ddf47a8d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:42:52,609::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:42:52,609::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:42:52,609::task::993::Storage.TaskManager.Task::(_decref) Task=`443fa593-e12b-4d74-bdcf-7af1ddf47a8d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:42:52,637::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:42:53,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:53,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919437 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:42:54,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:42:54,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:42:54,789::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000715039 s, 519 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:42:54,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640461 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:55,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:55,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914387 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:42:57,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:57,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915882 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:42:59,130::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:42:59,131::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:42:59,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:42:59,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913883 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:01,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:02,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092474 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:04,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:04,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921275 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:43:04,793::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:43:04,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:43:04,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654443 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:43:04,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000972503 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:06,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:43:07,650::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:43:08,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546809 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:43:09,141::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:43:09,142::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:43:10,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:10,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913363 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:12,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:12,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55866 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:43:13,444::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 13:43:14,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:43:14,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:43:14,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699734 s, 530 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:43:14,853::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689926 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:14,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:15,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561858 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:17,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:17,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913627 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:43:19,149::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:43:19,150::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:43:19,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:19,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912648 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:21,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:21,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914345 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:43:22,663::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:43:23,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:23,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912092 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:43:24,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:43:24,871::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665024 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:43:24,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:43:24,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649628 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:25,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:26,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913469 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:28,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:28,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091354 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:43:29,160::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:43:29,161::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:43:30,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:30,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913017 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:32,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:32,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510533 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:34,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:43:35,363::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:43:35,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:43:35,388::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00299873 s, 124 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:43:35,389::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656335 s, 797 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:35,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.59066 s, 1.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:37,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:37,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914758 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:43:37,675::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:43:39,170::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:43:39,171::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:43:39,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:40,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550882 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:42,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:42,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091485 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:43:43,483::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:43:44,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:44,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912463 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:43:45,400::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:43:45,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:43:45,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00488074 s, 76.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:43:45,425::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688239 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:46,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:46,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100478 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:48,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:48,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091323 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:43:49,180::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:43:49,182::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:43:50,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:51,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546934 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:43:52,678::task::595::Storage.TaskManager.Task::(_updateState) Task=`f37fb7da-f989-4a5a-be2f-1b92b16629ce`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:43:52,678::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:43:52,681::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:43:52,682::task::1191::Storage.TaskManager.Task::(prepare) Task=`f37fb7da-f989-4a5a-be2f-1b92b16629ce`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:43:52,682::task::595::Storage.TaskManager.Task::(_updateState) Task=`f37fb7da-f989-4a5a-be2f-1b92b16629ce`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:43:52,682::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:43:52,682::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:43:52,682::task::993::Storage.TaskManager.Task::(_decref) Task=`f37fb7da-f989-4a5a-be2f-1b92b16629ce`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:43:52,708::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:43:53,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:53,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558642 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:43:55,429::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:43:55,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:43:55,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00076214 s, 487 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:43:55,453::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000728634 s, 718 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:55,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:55,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912412 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:43:57,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:43:58,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912881 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:43:59,193::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:43:59,194::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:44:00,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:00,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912306 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:02,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:02,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913311 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:04,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:04,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912865 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:44:05,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:44:05,469::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:44:05,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00532032 s, 69.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:44:05,481::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653384 s, 800 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:06,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:06,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913624 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:44:07,720::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:44:08,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:08,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919115 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:44:09,201::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:44:09,202::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:44:10,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:10,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912967 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:12,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:12,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0935822 s, 10.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:44:13,516::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:44:14,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:14,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913551 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:44:15,756::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:44:15,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:44:15,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636768 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:44:15,783::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707051 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:16,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:17,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914535 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:19,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:44:19,214::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:44:19,215::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:44:19,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0948075 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:21,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:21,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911491 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:44:22,731::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:44:23,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:23,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555357 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:44:25,791::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:44:25,803::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:44:25,813::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00468788 s, 79.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:44:25,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000726164 s, 720 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:25,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:26,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091262 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:28,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:28,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913534 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:44:29,226::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:44:29,227::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:44:30,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:30,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912843 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:32,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:32,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543329 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:34,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:34,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912205 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:44:35,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:44:35,829::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630617 s, 588 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:44:35,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:44:35,845::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650263 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:36,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:37,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912662 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:44:37,743::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:44:39,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:39,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912704 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:44:39,238::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:44:39,239::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:44:41,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:41,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914205 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:43,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:43,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917636 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:44:43,548::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:44:45,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:45,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0998663 s, 10.3 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:44:45,847::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:44:45,862::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063687 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:44:45,867::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:44:45,883::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000617072 s, 848 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:47,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:47,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.090973 s, 11.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:44:49,246::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:44:49,247::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:44:49,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:49,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913326 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:51,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:52,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913522 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:44:52,746::task::595::Storage.TaskManager.Task::(_updateState) Task=`f8b988af-3bcf-48c2-abec-17556ddf57a7`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:44:52,746::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:44:52,749::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:44:52,749::task::1191::Storage.TaskManager.Task::(prepare) Task=`f8b988af-3bcf-48c2-abec-17556ddf57a7`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:44:52,750::task::595::Storage.TaskManager.Task::(_updateState) Task=`f8b988af-3bcf-48c2-abec-17556ddf57a7`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:44:52,750::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:44:52,750::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:44:52,750::task::993::Storage.TaskManager.Task::(_decref) Task=`f8b988af-3bcf-48c2-abec-17556ddf57a7`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:44:52,777::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:44:54,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:54,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914133 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:44:55,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:44:55,883::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657797 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:44:55,891::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:44:55,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679695 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:56,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:56,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556166 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:44:58,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:44:58,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0942963 s, 10.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:44:59,258::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:44:59,259::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:45:00,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:00,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913391 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:03,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:03,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547211 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:05,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:05,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911171 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:45:05,887::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:45:05,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683147 s, 543 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:45:05,906::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:45:05,917::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642771 s, 814 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:07,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:45:07,800::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:45:07,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913529 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:45:09,266::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:45:09,268::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:45:09,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:09,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914591 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:11,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:12,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914176 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:45:13,580::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:45:14,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:14,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914683 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:45:15,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:45:15,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00384985 s, 96.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:45:15,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:45:15,945::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00553469 s, 94.5 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:16,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:16,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968556 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:18,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:18,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913298 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:45:19,274::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:45:19,275::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:45:20,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:20,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0997101 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:22,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:45:22,814::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:45:22,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912468 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:24,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:25,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556454 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:45:25,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:45:25,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667499 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:45:25,952::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:45:25,963::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692085 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:27,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:28,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557571 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:45:29,286::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:45:29,287::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:45:30,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:30,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914753 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:32,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:32,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912807 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:34,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:34,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911716 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:45:35,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:45:35,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684143 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:45:35,967::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:45:35,979::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647171 s, 808 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:36,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:36,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09166 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:45:37,826::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:45:38,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:39,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557511 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:45:39,298::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:45:39,300::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:45:41,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:41,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914002 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:43,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:43,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091369 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:45:43,617::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:45:45,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:45,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913099 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:45:45,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:45:45,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00081732 s, 454 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:45:45,985::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:45:45,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064607 s, 810 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:47,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:47,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912586 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:45:49,310::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:45:49,311::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:45:49,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:49,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913371 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:51,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:51,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912719 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:45:52,829::task::595::Storage.TaskManager.Task::(_updateState) Task=`577640c3-16fc-4cda-adc8-88c9b79b0067`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:45:52,829::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:45:52,832::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:45:52,832::task::1191::Storage.TaskManager.Task::(prepare) Task=`577640c3-16fc-4cda-adc8-88c9b79b0067`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:45:52,832::task::595::Storage.TaskManager.Task::(_updateState) Task=`577640c3-16fc-4cda-adc8-88c9b79b0067`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:45:52,833::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:45:52,833::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:45:52,833::task::993::Storage.TaskManager.Task::(_decref) Task=`577640c3-16fc-4cda-adc8-88c9b79b0067`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:45:52,858::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:45:53,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:53,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913396 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:45:55,978::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:45:55,990::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673111 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:55,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:45:56,561::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:56,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.568508 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:45:56,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00221103 s, 237 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:45:58,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:45:58,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924051 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:45:59,314::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:45:59,316::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:46:00,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:00,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913525 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:02,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:02,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913804 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:04,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:05,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091235 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:46:05,993::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:46:06,005::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663207 s, 559 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:46:06,583::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:46:06,594::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698651 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:07,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:07,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091269 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:46:07,870::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:46:09,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:09,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913496 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:46:09,326::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:46:09,327::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:46:11,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:11,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922817 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:13,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:13,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914228 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:46:13,649::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:46:15,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:15,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967165 s, 10.6 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:46:16,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:46:16,024::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681634 s, 544 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:46:16,598::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:46:16,610::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000800505 s, 653 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:17,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:17,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091324 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:46:19,338::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:46:19,339::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:46:19,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:19,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915867 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:21,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:22,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913362 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:46:22,882::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:46:24,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:24,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914316 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:46:26,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:46:26,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000730236 s, 508 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:26,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:26,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09131 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:46:26,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:46:26,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653114 s, 801 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:28,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:28,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913581 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:46:29,346::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:46:29,347::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:46:30,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:30,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912907 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:32,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:32,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913143 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:34,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:34,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.090926 s, 11.3 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:46:36,046::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:46:36,057::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000609284 s, 609 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:46:36,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:46:36,642::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647206 s, 808 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:36,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:36,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912332 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:46:37,894::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:46:38,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:39,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913272 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:46:39,358::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:46:39,359::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:46:41,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:41,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920288 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:43,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:43,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911532 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:46:43,680::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:46:45,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:45,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913199 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:46:46,063::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:46:46,076::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686121 s, 541 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:46:46,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:46:46,657::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067228 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:47,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:47,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914315 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:46:49,370::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:46:49,371::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:46:49,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:49,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.114446 s, 8.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:51,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:51,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912906 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:46:52,897::task::595::Storage.TaskManager.Task::(_updateState) Task=`51ca4df4-1670-48ca-b98a-f0ea92dd5c86`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:46:52,897::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:46:52,900::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:46:52,901::task::1191::Storage.TaskManager.Task::(prepare) Task=`51ca4df4-1670-48ca-b98a-f0ea92dd5c86`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:46:52,901::task::595::Storage.TaskManager.Task::(_updateState) Task=`51ca4df4-1670-48ca-b98a-f0ea92dd5c86`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:46:52,901::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:46:52,901::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:46:52,901::task::993::Storage.TaskManager.Task::(_decref) Task=`51ca4df4-1670-48ca-b98a-f0ea92dd5c86`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:46:52,928::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:46:53,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:53,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.090995 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:55,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:56,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912171 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:46:56,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:46:56,096::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000723512 s, 513 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:46:56,660::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:46:56,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683198 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:46:58,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:46:58,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915867 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:46:59,382::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:46:59,383::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:47:00,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:00,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912906 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:02,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:02,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912087 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:04,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:04,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913176 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:47:06,099::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:47:06,111::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674756 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:06,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:06,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913558 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:47:06,674::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:47:06,690::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:47:06,692::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:47:06,695::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:47:06,696::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:47:06,696::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:47:06,696::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:47:06,697::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:47:06,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000622768 s, 840 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:47:07,940::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:47:08,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:08,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913148 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:47:09,394::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:47:09,395::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:47:10,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:10,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914207 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:12,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:12,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913789 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:47:13,715::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:47:14,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:15,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914487 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:47:16,117::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:47:16,130::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618738 s, 600 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:47:16,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:47:16,724::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660269 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:17,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:17,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09137 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:19,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:19,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091239 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:47:19,401::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:47:19,402::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:47:21,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:21,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913322 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:47:22,952::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:47:23,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:23,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912613 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:25,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:26,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545228 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:47:26,136::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:47:26,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643474 s, 577 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:47:26,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:47:26,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000718009 s, 728 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:28,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:28,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0931677 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:47:29,412::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:47:29,414::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:47:30,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:30,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091295 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:32,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:32,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091477 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:34,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:34,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912935 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:47:36,150::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:47:36,176::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:47:36,178::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:47:36,181::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:47:36,182::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:47:36,182::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:47:36,183::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:47:36,184::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:47:36,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068161 s, 544 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:36,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:36,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912177 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:47:36,745::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:47:36,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664662 s, 787 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:47:37,966::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:47:38,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:38,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913032 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:47:39,422::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:47:39,423::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:47:40,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:40,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913371 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:42,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:43,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549406 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:47:43,763::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:47:45,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:45,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101295 s, 10.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:47:46,200::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:47:46,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00077753 s, 477 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:47:46,761::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:47:46,773::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684485 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:47,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:47,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913039 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:47:49,431::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:47:49,432::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:47:49,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:50,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914536 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:52,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:52,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557246 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:47:52,970::task::595::Storage.TaskManager.Task::(_updateState) Task=`fd9fde47-c3cf-40d7-83bd-a3c88430c156`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:47:52,970::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:47:52,973::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:47:52,973::task::1191::Storage.TaskManager.Task::(prepare) Task=`fd9fde47-c3cf-40d7-83bd-a3c88430c156`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:47:52,973::task::595::Storage.TaskManager.Task::(_updateState) Task=`fd9fde47-c3cf-40d7-83bd-a3c88430c156`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:47:52,973::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:47:52,973::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:47:52,974::task::993::Storage.TaskManager.Task::(_decref) Task=`fd9fde47-c3cf-40d7-83bd-a3c88430c156`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:47:52,998::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:47:54,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:54,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915272 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:47:56,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:47:56,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066179 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:56,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:47:57,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:57,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912288 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:47:57,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00534531 s, 97.8 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:47:59,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:47:59,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912213 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:47:59,458::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:47:59,462::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:48:01,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:01,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912947 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:03,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:03,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910425 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:05,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:05,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912527 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:48:06,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:48:06,244::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671117 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:48:07,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:48:07,329::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697313 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:07,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:08,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921125 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:48:08,015::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:48:09,462::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:48:09,463::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:48:10,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:10,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914937 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:12,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:12,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914036 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:48:13,802::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:48:14,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:14,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091379 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:48:16,250::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:48:16,264::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628653 s, 590 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:16,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:16,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999746 s, 10.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:48:17,333::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:48:17,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696182 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:18,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:18,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911994 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:48:19,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:48:19,476::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:48:20,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:20,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911798 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:22,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:22,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911431 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:48:23,030::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:48:24,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:25,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912344 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:48:26,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:48:26,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067108 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:27,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:27,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914692 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:48:27,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:48:27,360::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065189 s, 802 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:29,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:48:29,483::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:48:29,484::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:48:29,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542759 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:31,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:31,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912384 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:33,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:33,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09141 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:35,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:36,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914156 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:48:36,285::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:48:36,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000707912 s, 524 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:48:37,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:48:37,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675463 s, 774 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:48:38,042::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:48:38,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:38,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556875 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:48:39,492::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:48:39,493::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:48:40,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:41,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543427 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:43,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:43,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543952 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:48:43,840::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:48:45,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:45,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0987873 s, 10.4 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:48:46,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:48:46,320::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00660972 s, 56.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:48:47,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:48:47,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000733174 s, 713 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:48,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:48,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914035 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:48:49,504::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:48:49,505::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:48:50,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:50,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913767 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:52,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:52,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916732 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:48:53,045::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf71ad76-4ec1-4f59-949f-2fe657df4a3a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:48:53,045::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:48:53,049::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:48:53,049::task::1191::Storage.TaskManager.Task::(prepare) Task=`bf71ad76-4ec1-4f59-949f-2fe657df4a3a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:48:53,049::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf71ad76-4ec1-4f59-949f-2fe657df4a3a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:48:53,049::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:48:53,049::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:48:53,050::task::993::Storage.TaskManager.Task::(_decref) Task=`bf71ad76-4ec1-4f59-949f-2fe657df4a3a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:48:53,076::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:48:54,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:54,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914158 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:48:56,325::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:48:56,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663911 s, 559 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:56,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:56,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917483 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:48:57,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:48:57,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000731526 s, 715 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:48:58,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:48:58,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913265 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:48:59,517::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:48:59,518::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:49:00,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:00,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913538 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:02,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:03,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551591 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:05,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:06,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55903 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:49:06,509::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:49:06,521::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665637 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:49:07,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:49:07,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680681 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:08,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:49:08,090::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:49:08,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543627 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:49:09,529::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:49:09,530::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:49:10,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:11,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550095 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:13,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:13,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921266 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:49:13,870::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:49:15,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:15,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914462 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:49:16,526::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:49:16,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626256 s, 592 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:17,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:49:17,549::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:49:17,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065734 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:17,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.148509 s, 6.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:49:19,541::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:49:19,542::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:49:19,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:19,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912733 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:21,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:22,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545111 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:49:23,102::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:49:24,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:24,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913385 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:26,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:26,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912516 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:49:26,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:49:26,553::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622392 s, 596 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:49:27,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:49:27,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700994 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:28,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:28,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913599 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:49:29,553::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:49:29,554::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:49:30,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:30,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912871 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:32,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:32,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914263 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:34,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:34,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913378 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:49:36,556::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:49:36,568::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679316 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:36,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:37,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913419 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:49:37,588::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:49:37,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629435 s, 831 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:49:38,113::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:49:39,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:39,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912323 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:49:39,565::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:49:39,567::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:49:41,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:41,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913047 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:43,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:43,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913552 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:49:43,902::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:49:45,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:45,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0985723 s, 10.4 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:49:46,572::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:49:46,583::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069115 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:49:47,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:47,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:49:47,700::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00233879 s, 224 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:47,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910815 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:49:49,577::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:49:49,579::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:49:49,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:50,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.580701 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:52,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:52,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913204 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:49:53,117::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5884a9b-36fb-4c2a-b5b9-2adc224b363f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:49:53,117::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:49:53,120::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:49:53,120::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5884a9b-36fb-4c2a-b5b9-2adc224b363f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:49:53,121::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5884a9b-36fb-4c2a-b5b9-2adc224b363f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:49:53,121::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:49:53,121::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:49:53,121::task::993::Storage.TaskManager.Task::(_decref) Task=`a5884a9b-36fb-4c2a-b5b9-2adc224b363f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:49:53,148::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:49:54,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:54,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921619 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:49:56,586::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:49:56,598::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668729 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:56,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:56,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914566 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:49:57,704::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:49:57,716::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687965 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:49:58,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:49:58,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920595 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:49:59,590::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:49:59,591::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:50:00,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:00,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919198 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:03,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:03,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916923 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:05,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:05,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915254 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:50:06,601::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:50:06,613::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674733 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:07,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:07,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913458 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:50:07,719::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:50:07,731::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689761 s, 758 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:50:08,161::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:50:09,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:09,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913087 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:50:09,598::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:50:09,599::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:50:11,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:11,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911721 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:13,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:13,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09202 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:50:13,934::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:50:15,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:15,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0972322 s, 10.5 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:50:16,616::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:50:16,628::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649202 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:50:17,735::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:50:17,746::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000623824 s, 838 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:17,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:18,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913499 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:50:19,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:50:19,610::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:50:20,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:20,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557329 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:22,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:22,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912562 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:50:23,174::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:50:24,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:24,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913603 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:50:26,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:50:26,644::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000616668 s, 602 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:26,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:26,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914305 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:50:27,750::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:50:27,762::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649688 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:28,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:29,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914785 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:50:29,619::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:50:29,621::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:50:31,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:31,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.533056 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:33,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:33,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913171 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:35,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:35,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912524 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:50:36,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:50:36,658::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643343 s, 577 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:50:37,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:50:37,777::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000644462 s, 812 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:37,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:38,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912767 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:50:38,188::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:50:39,631::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:50:39,632::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:50:40,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:40,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916504 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:42,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:42,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091363 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:50:43,967::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:50:44,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:44,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914745 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:46,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:46,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0947776 s, 10.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:50:46,662::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:50:46,674::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675231 s, 549 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:50:47,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:50:47,793::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000722571 s, 724 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:48,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:48,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911868 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:50:49,644::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:50:49,645::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:50:50,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:50,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912871 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:52,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:50:53,191::task::595::Storage.TaskManager.Task::(_updateState) Task=`38cc4ca5-2ad0-4999-b4a2-811773c9ee80`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:50:53,192::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:50:53,296::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:50:53,297::task::1191::Storage.TaskManager.Task::(prepare) Task=`38cc4ca5-2ad0-4999-b4a2-811773c9ee80`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:50:53,297::task::595::Storage.TaskManager.Task::(_updateState) Task=`38cc4ca5-2ad0-4999-b4a2-811773c9ee80`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:50:53,298::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:50:53,299::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:50:53,299::task::993::Storage.TaskManager.Task::(_decref) Task=`38cc4ca5-2ad0-4999-b4a2-811773c9ee80`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 13:50:53,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.508526 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:50:53,329::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:50:55,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:55,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545192 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:50:56,677::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:50:56,689::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680396 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:50:57,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:50:57,808::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683596 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:50:57,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:50:57,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914338 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:50:59,653::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:50:59,655::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:51:00,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:00,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912592 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:02,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:02,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910513 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:04,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:04,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912161 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:06,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:06,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912551 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:51:06,692::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:51:06,704::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000695097 s, 534 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:51:07,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:51:07,823::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684098 s, 765 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:51:08,341::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:51:08,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:08,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912322 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:51:09,666::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:51:09,667::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:51:10,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:10,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913986 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:12,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:12,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916291 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:51:14,001::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:51:14,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:14,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913677 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:51:16,707::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:51:16,719::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621629 s, 597 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:16,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:17,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542885 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:51:17,827::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:51:17,839::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661644 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:19,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:19,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914595 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:51:19,678::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:51:19,679::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:51:21,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:22,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559492 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:51:23,355::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:51:24,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:24,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0941296 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:26,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:51:27,116::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:51:27,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663993 s, 559 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:27,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561148 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:51:27,842::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:51:27,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720962 s, 725 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:29,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:29,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914712 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:51:29,687::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:51:29,688::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:51:31,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:31,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091423 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:33,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:33,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54267 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:35,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:36,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915054 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:51:37,135::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:51:37,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668648 s, 555 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:51:37,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:51:37,871::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068282 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:38,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:38,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.13368 s, 7.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:51:38,367::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:51:39,699::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:51:39,700::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:51:40,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:40,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551328 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:42,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:42,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914906 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:51:44,034::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:51:44,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:45,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559859 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:51:47,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:51:47,162::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693999 s, 535 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:47,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:47,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913269 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:51:47,875::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:51:47,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689584 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:49,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:51:49,712::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:51:49,713::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:51:49,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913199 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:51,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:51,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917227 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:51:53,370::task::595::Storage.TaskManager.Task::(_updateState) Task=`e0a038b7-077b-4ee1-9d6f-93478e14844d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:51:53,370::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:51:53,373::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:51:53,373::task::1191::Storage.TaskManager.Task::(prepare) Task=`e0a038b7-077b-4ee1-9d6f-93478e14844d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:51:53,373::task::595::Storage.TaskManager.Task::(_updateState) Task=`e0a038b7-077b-4ee1-9d6f-93478e14844d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:51:53,373::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:51:53,373::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:51:53,374::task::993::Storage.TaskManager.Task::(_decref) Task=`e0a038b7-077b-4ee1-9d6f-93478e14844d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:51:53,399::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:51:53,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:53,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913915 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:55,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:56,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913445 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:51:57,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:51:57,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645291 s, 575 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:51:57,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:51:57,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694735 s, 753 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:51:58,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:51:58,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557426 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:51:59,724::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:51:59,725::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:52:00,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:00,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914401 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:02,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:03,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55907 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:05,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:05,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914698 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:52:07,181::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:52:07,192::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654371 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:07,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:52:07,904::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Dummy-49::DEBUG::2014-07-14 13:52:08,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544703 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:52:08,073::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:52:08,075::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:52:08,079::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:52:08,083::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:52:08,083::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:52:08,084::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:52:08,087::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:52:08,101::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699313 s, 748 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:52:08,411::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:52:09,735::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:52:09,736::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:52:10,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:10,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915219 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:12,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:12,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912311 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:52:14,066::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:52:14,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:14,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913428 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:16,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:16,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910167 s, 11.3 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:52:17,196::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:52:17,208::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635315 s, 584 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:52:18,111::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:52:18,123::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680663 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:18,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:19,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545346 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:52:19,743::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:52:19,744::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:52:21,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:21,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912987 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:23,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:52:23,423::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:52:23,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55795 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:25,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:25,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913935 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:52:27,212::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:52:27,223::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006054 s, 613 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:27,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:28,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914302 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:52:28,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:52:28,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697277 s, 750 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:52:29,755::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:52:29,756::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:52:30,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:30,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914353 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:32,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:32,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914194 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:34,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:34,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914218 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:36,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:36,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913508 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:52:37,225::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:52:37,239::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 13:52:37,240::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 13:52:37,243::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:52:37,245::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:52:37,245::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:52:37,245::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 13:52:37,246::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:52:37,257::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636768 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:52:38,142::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:52:38,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659529 s, 793 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:52:38,435::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:52:38,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:38,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915182 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:52:39,767::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:52:39,769::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:52:40,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:40,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914356 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:42,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:42,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913375 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:52:44,099::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:52:44,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:45,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091536 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:47,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:47,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913557 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:52:47,261::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:52:47,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.556224 s, 0.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:52:48,158::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:52:48,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000708606 s, 738 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:49,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:49,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091322 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:52:49,776::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:52:49,777::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:52:51,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:51,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915164 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:53,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:52:53,438::task::595::Storage.TaskManager.Task::(_updateState) Task=`13e5a680-8ba9-4dcd-a5f6-21d0f0743fa5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:52:53,438::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:52:53,494::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:52:53,495::task::1191::Storage.TaskManager.Task::(prepare) Task=`13e5a680-8ba9-4dcd-a5f6-21d0f0743fa5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:52:53,496::task::595::Storage.TaskManager.Task::(_updateState) Task=`13e5a680-8ba9-4dcd-a5f6-21d0f0743fa5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:52:53,497::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:52:53,498::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:52:53,499::task::993::Storage.TaskManager.Task::(_decref) Task=`13e5a680-8ba9-4dcd-a5f6-21d0f0743fa5`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 13:52:53,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964464 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:52:53,526::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:52:55,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:55,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913851 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:57,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:52:57,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912864 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:52:57,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:52:57,843::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677604 s, 548 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:52:58,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:52:58,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700853 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:52:59,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:52:59,787::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:52:59,789::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:52:59,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914834 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:01,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:01,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913393 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:03,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:04,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912967 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:06,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:06,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091301 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:53:07,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:53:07,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675608 s, 549 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:53:08,189::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:53:08,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691289 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:08,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:53:08,539::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:53:08,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543308 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:53:09,799::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:53:09,800::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:53:10,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:10,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914864 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:12,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:13,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550312 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:53:14,130::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:53:15,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:15,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0998152 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:17,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:17,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091304 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:53:17,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:53:17,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649385 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:53:18,204::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:53:18,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676621 s, 773 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:53:19,811::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:53:19,812::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:53:19,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:19,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913021 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:21,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:22,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912724 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:53:23,552::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:53:24,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:24,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913211 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:26,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:26,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912537 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:53:27,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:53:27,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654435 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:53:28,220::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:53:28,231::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668982 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:28,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:28,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54339 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:53:29,823::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:53:29,824::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:53:30,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:31,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.133104 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:33,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:33,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914943 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:35,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:35,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551034 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:37,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:37,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913572 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:53:37,892::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:53:37,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000712501 s, 521 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:53:38,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:53:38,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000638398 s, 819 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:53:38,565::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:53:39,835::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:53:39,837::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:53:39,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:39,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913831 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:41,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:42,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921642 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:44,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:53:44,164::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:53:44,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543733 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:46,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:47,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551135 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:53:47,907::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:53:47,919::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662672 s, 560 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:53:48,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:53:48,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681581 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:49,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:49,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913079 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:53:49,848::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:53:49,849::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:53:51,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:51,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914051 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:53,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:53:53,569::task::595::Storage.TaskManager.Task::(_updateState) Task=`a51605d8-3e9a-48bc-b5eb-e5085155950d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:53:53,569::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:53:53,984::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:53:53,985::task::1191::Storage.TaskManager.Task::(prepare) Task=`a51605d8-3e9a-48bc-b5eb-e5085155950d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:53:53,986::task::595::Storage.TaskManager.Task::(_updateState) Task=`a51605d8-3e9a-48bc-b5eb-e5085155950d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:53:53,986::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:53:53,987::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:53:53,988::task::993::Storage.TaskManager.Task::(_decref) Task=`a51605d8-3e9a-48bc-b5eb-e5085155950d`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 13:53:53,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.513259 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:53:54,016::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:53:56,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:56,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915229 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:53:57,922::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:53:57,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671496 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:53:58,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:53:58,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091396 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:53:58,266::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:53:58,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000714438 s, 732 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:53:59,856::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:53:59,857::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:54:00,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:00,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09132 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:02,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:02,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913822 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:04,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:04,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909841 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:06,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:06,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912864 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:54:07,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:54:07,950::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000761257 s, 487 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:54:08,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:54:08,293::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640122 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:08,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:08,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914248 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:54:09,029::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:54:09,868::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:54:09,870::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:54:10,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:10,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916005 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:12,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:13,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914712 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:54:14,197::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:54:15,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:15,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915023 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:17,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:17,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923746 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:54:17,954::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:54:17,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620241 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:54:18,298::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:54:18,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663779 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:19,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:19,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911746 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:54:19,876::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:54:19,877::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:54:21,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:21,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913168 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:23,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:23,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913963 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:54:24,048::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:54:25,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:25,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914995 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:27,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:27,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09148 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:54:27,969::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:54:27,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618933 s, 599 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:54:28,316::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:54:28,328::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657715 s, 795 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:29,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:54:29,896::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:54:29,897::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:54:29,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0908836 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:31,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:32,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915445 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:34,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:34,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912927 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:36,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:36,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914191 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:54:37,985::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:54:37,996::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000717741 s, 517 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:54:38,332::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:54:38,344::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632427 s, 827 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:38,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:38,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914182 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:54:39,059::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:54:39,906::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:54:39,907::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:54:40,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:40,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913319 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:42,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:42,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912903 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:54:44,229::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:54:44,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:44,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915155 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:46,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:46,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913762 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:54:48,000::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:54:48,012::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697375 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:54:48,348::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:54:48,360::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693695 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:48,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:49,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912568 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:54:49,916::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:54:49,917::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:54:51,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:51,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555682 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:53,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:53,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912567 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:54:54,062::task::595::Storage.TaskManager.Task::(_updateState) Task=`34f11668-23e4-4f9c-8bbf-9de3bdc5c9b6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:54:54,063::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:54:54,065::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:54:54,065::task::1191::Storage.TaskManager.Task::(prepare) Task=`34f11668-23e4-4f9c-8bbf-9de3bdc5c9b6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:54:54,065::task::595::Storage.TaskManager.Task::(_updateState) Task=`34f11668-23e4-4f9c-8bbf-9de3bdc5c9b6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:54:54,065::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:54:54,066::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:54:54,066::task::993::Storage.TaskManager.Task::(_decref) Task=`34f11668-23e4-4f9c-8bbf-9de3bdc5c9b6`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:54:54,091::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:54:55,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:54:55,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913331 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:57,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:54:58,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:54:58,410::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656753 s, 796 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:54:58,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:54:58,445::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00123712 s, 300 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:54:58,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.572144 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:54:59,926::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:54:59,927::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:55:00,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:01,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558333 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:03,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:03,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915346 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:05,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:05,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558527 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:07,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:07,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913991 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:55:08,448::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:55:08,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000790403 s, 662 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:55:08,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:55:08,475::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000725283 s, 512 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:55:09,103::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:55:09,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:55:09,939::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:55:09,940::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:55:09,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914662 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:11,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:12,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920395 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:14,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:14,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915344 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:55:14,261::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:55:16,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:16,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999103 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:18,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:55:18,487::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:55:18,499::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:55:18,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000485705 s, 1.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:55:18,579::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000790731 s, 469 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:18,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102794 s, 10.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:55:19,946::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:55:19,947::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:55:20,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:20,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915239 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:22,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:23,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.513666 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:55:24,115::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:55:25,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:25,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914113 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:27,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:27,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091276 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:55:28,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:55:28,598::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:55:28,609::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00562557 s, 65.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:55:28,611::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000634816 s, 824 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:29,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:29,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913253 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:55:29,956::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:55:29,957::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:55:31,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:31,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914733 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:33,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:33,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913768 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:35,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:36,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915179 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:38,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:38,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915475 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:55:38,616::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:55:38,627::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:55:38,643::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636485 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:55:38,645::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660169 s, 792 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:55:39,128::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:55:39,967::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:55:39,968::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:55:40,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:40,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091435 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:42,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:42,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913789 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:55:44,294::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:55:44,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:44,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915526 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:46,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:46,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912237 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:55:48,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:55:48,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:55:48,662::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000609853 s, 608 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:55:48,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000808463 s, 647 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:48,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:48,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912946 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:55:49,975::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:55:49,976::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:55:50,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:51,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557897 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:53,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:53,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104638 s, 9.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:55:54,131::task::595::Storage.TaskManager.Task::(_updateState) Task=`b34428e1-b1e3-40a2-8437-13ce40813840`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:55:54,132::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:55:54,134::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:55:54,135::task::1191::Storage.TaskManager.Task::(prepare) Task=`b34428e1-b1e3-40a2-8437-13ce40813840`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:55:54,135::task::595::Storage.TaskManager.Task::(_updateState) Task=`b34428e1-b1e3-40a2-8437-13ce40813840`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:55:54,135::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:55:54,135::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:55:54,135::task::993::Storage.TaskManager.Task::(_decref) Task=`b34428e1-b1e3-40a2-8437-13ce40813840`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:55:54,162::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:55:55,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:55,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915015 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:57,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:55:57,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915476 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:55:58,679::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:55:58,690::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:55:58,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00189951 s, 195 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:55:58,703::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0010325 s, 507 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:55:59,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:55:59,983::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:55:59,984::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:56:00,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549716 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:02,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:02,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915759 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:04,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:05,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547049 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:07,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:07,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913675 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:56:08,706::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:56:08,718::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:56:08,729::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00559425 s, 66.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:56:08,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000719924 s, 726 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:56:09,175::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:56:09,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:09,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915703 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:56:09,995::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:56:09,996::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:56:11,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:11,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913101 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:13,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:13,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914311 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:56:14,324::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:56:15,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:15,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0969995 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:17,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:17,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915039 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:56:18,735::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:56:18,746::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:56:18,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000594816 s, 624 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:56:18,759::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000849907 s, 615 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:19,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:56:20,007::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:56:20,009::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:56:20,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912998 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:22,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:22,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919145 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:56:24,188::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:56:24,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:24,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542274 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:26,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:26,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913137 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:56:28,763::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:56:28,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:56:28,789::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637516 s, 582 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:56:28,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000785927 s, 665 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:28,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:28,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917304 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:56:30,020::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:56:30,021::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:56:30,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:31,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914694 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:33,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:33,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914192 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:35,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:35,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914364 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:37,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:37,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913322 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:56:38,793::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:56:38,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067349 s, 551 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:56:38,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:56:38,819::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720201 s, 726 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:56:39,200::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:56:39,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:39,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913951 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:56:40,032::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:56:40,033::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:56:41,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:41,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091361 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:43,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:43,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915212 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:56:44,359::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:56:45,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:46,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554024 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:48,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:48,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919334 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:56:48,819::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:56:48,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000765148 s, 485 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:56:48,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:56:48,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696546 s, 751 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:56:50,043::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:56:50,044::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:56:50,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:50,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914677 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:52,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:53,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544775 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:56:54,204::task::595::Storage.TaskManager.Task::(_updateState) Task=`4867286b-2509-4d0a-af00-159c4e533b29`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:56:54,204::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:56:54,207::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:56:54,207::task::1191::Storage.TaskManager.Task::(prepare) Task=`4867286b-2509-4d0a-af00-159c4e533b29`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:56:54,207::task::595::Storage.TaskManager.Task::(_updateState) Task=`4867286b-2509-4d0a-af00-159c4e533b29`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:56:54,207::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:56:54,207::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:56:54,208::task::993::Storage.TaskManager.Task::(_decref) Task=`4867286b-2509-4d0a-af00-159c4e533b29`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:56:54,234::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:56:55,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:55,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913405 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:56:57,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:56:58,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545323 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:56:58,847::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:56:58,860::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000778728 s, 476 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:56:58,860::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:56:58,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066577 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:00,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:57:00,056::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:57:00,057::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:57:00,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914226 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:02,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:02,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916062 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:04,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:04,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913492 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:06,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:06,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918039 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:08,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 13:57:08,876::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 13:57:08,896::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:57:08,911::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618955 s, 599 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:08,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.408865 s, 2.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:57:08,947::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 13:57:08,949::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:57:08,953::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 13:57:08,956::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 13:57:08,956::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 13:57:08,957::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 13:57:08,959::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:57:08,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000610785 s, 856 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:57:09,247::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:57:10,068::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:57:10,069::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:57:10,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:11,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914806 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:13,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:13,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916343 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:57:14,391::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:57:15,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:15,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914562 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:17,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:17,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091365 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:57:18,921::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:57:18,933::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00061358 s, 605 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:57:18,980::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:57:18,991::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000627291 s, 834 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:19,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:19,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.534212 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:57:20,077::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:57:20,078::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:57:22,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:22,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913112 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:24,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:24,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914307 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:57:24,260::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:57:26,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:26,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920914 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:28,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:28,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913835 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:57:28,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:57:28,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000624951 s, 594 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:57:28,995::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:57:29,006::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691184 s, 757 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:57:30,089::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:57:30,090::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:57:30,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:30,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912757 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:32,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:32,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915883 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:34,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:34,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913932 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:36,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:36,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913419 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:57:38,950::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Dummy-49::DEBUG::2014-07-14 13:57:38,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:57:38,976::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 13:57:39,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:57:39,065::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 13:57:39,067::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673256 s, 777 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:57:39,071::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 13:57:39,079::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 13:57:39,081::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 13:57:39,081::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Dummy-49::DEBUG::2014-07-14 13:57:39,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.117907 s, 8.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:57:39,086::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:57:39,106::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00612796 s, 60.5 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:57:39,273::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:57:40,100::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:57:40,101::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:57:41,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:41,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919348 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:43,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:43,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918415 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:57:44,424::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:57:45,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:45,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921643 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:47,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:47,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912532 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:57:49,078::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:57:49,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716218 s, 730 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:57:49,111::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:57:49,123::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676997 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:49,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:49,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091316 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:57:50,108::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:57:50,109::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:57:51,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:52,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552024 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:57:54,277::task::595::Storage.TaskManager.Task::(_updateState) Task=`cdb1dd2e-996d-43db-ad0b-b61c9b1f8657`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:57:54,277::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:57:54,280::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:57:54,280::task::1191::Storage.TaskManager.Task::(prepare) Task=`cdb1dd2e-996d-43db-ad0b-b61c9b1f8657`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:57:54,280::task::595::Storage.TaskManager.Task::(_updateState) Task=`cdb1dd2e-996d-43db-ad0b-b61c9b1f8657`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:57:54,280::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:57:54,280::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:57:54,281::task::993::Storage.TaskManager.Task::(_decref) Task=`cdb1dd2e-996d-43db-ad0b-b61c9b1f8657`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 13:57:54,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:57:54,317::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:57:54,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915437 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:56,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:56,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917114 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:57:58,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:57:58,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915113 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:57:59,093::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:57:59,105::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000709806 s, 737 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:57:59,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:57:59,137::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681562 s, 544 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:58:00,121::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:58:00,122::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:58:00,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:00,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913804 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:02,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:02,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091565 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:04,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:05,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910938 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:07,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:08,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913037 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:58:09,108::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:58:09,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692675 s, 755 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:58:09,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:58:09,151::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643868 s, 576 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:58:09,330::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:58:10,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 13:58:10,132::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:58:10,133::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:58:10,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917588 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:12,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:12,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921925 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:14,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 13:58:14,456::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:58:14,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555837 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:16,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:17,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913507 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:19,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 13:58:19,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:58:19,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:58:19,237::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632834 s, 586 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:58:19,238::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062188 s, 841 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:19,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.165375 s, 6.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:58:20,144::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:58:20,145::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:58:21,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:21,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915146 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:23,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:23,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911727 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:58:24,343::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:58:25,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:25,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913119 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:27,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:27,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912551 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:58:29,243::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:58:29,254::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:58:29,265::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00522388 s, 71.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:58:29,266::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657232 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:29,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:29,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914375 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:58:30,156::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:58:30,157::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:58:31,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:31,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913371 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:33,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:34,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913081 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:36,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:36,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915323 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:38,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:38,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554094 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:58:39,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:58:39,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:58:39,298::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00056886 s, 652 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:58:39,300::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000717993 s, 728 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:58:39,356::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 13:58:40,168::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:58:40,169::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:58:40,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:40,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091594 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:42,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:42,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911863 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:58:44,489::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:58:45,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:45,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.522367 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:47,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:47,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912757 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:58:49,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:58:49,314::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:58:49,330::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691506 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:58:49,331::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631881 s, 828 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:49,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:49,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912999 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:58:50,180::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:58:50,181::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:58:51,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:52,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557468 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:58:54,359::task::595::Storage.TaskManager.Task::(_updateState) Task=`527ebc57-f2ac-496f-97a1-2d4271788ca0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:58:54,360::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:58:54,362::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:58:54,362::task::1191::Storage.TaskManager.Task::(prepare) Task=`527ebc57-f2ac-496f-97a1-2d4271788ca0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:58:54,363::task::595::Storage.TaskManager.Task::(_updateState) Task=`527ebc57-f2ac-496f-97a1-2d4271788ca0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:58:54,363::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:58:54,363::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:58:54,363::task::993::Storage.TaskManager.Task::(_decref) Task=`527ebc57-f2ac-496f-97a1-2d4271788ca0`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:58:54,389::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:58:54,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:54,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914314 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:56,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:56,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913552 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:58:58,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:58:58,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912868 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:58:59,334::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:58:59,346::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:58:59,361::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673421 s, 551 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:58:59,363::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00106254 s, 492 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:59:00,192::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:59:00,194::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:59:00,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:00,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914141 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:02,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:03,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911355 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:05,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:05,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920659 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:07,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:07,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914668 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:09,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 13:59:09,402::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 13:59:09,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:59:09,436::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:59:09,460::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000798238 s, 655 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:59:09,464::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622957 s, 596 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:09,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.12995 s, 7.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:59:10,206::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:59:10,207::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:59:11,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:11,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913102 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:13,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:13,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913766 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:59:14,523::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:59:15,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:15,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.105753 s, 9.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:17,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:18,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913182 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:59:19,471::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:59:19,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:59:19,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656088 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:59:19,495::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621201 s, 842 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:20,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:20,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912042 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:59:20,218::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:59:20,219::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:59:22,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:22,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.509857 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:59:24,414::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:59:24,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:25,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50758 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:27,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:27,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912043 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:29,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:29,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913133 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:59:29,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:59:29,503::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:59:29,524::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639074 s, 581 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:59:29,526::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000717686 s, 729 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:59:30,230::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:59:30,232::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:59:31,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:31,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914232 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:33,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:33,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913581 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:35,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:35,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912852 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:37,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:37,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132756 s, 7.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:59:39,426::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 13:59:39,530::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:59:39,541::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:59:39,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662711 s, 560 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:59:39,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676654 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:39,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:40,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913807 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:59:40,243::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:59:40,244::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:59:42,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:42,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912858 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:44,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:44,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132737 s, 7.7 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 13:59:44,562::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 13:59:46,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:46,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09672 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:48,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:48,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091673 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:59:49,564::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:59:49,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:59:49,578::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000666936 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:59:49,590::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713217 s, 733 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 13:59:50,254::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 13:59:50,255::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 13:59:50,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:50,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916719 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:52,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:52,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914132 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 13:59:54,429::task::595::Storage.TaskManager.Task::(_updateState) Task=`bfd88fc8-0608-4acd-b25a-7eee8ba89e15`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 13:59:54,430::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 13:59:54,432::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:59:54,432::task::1191::Storage.TaskManager.Task::(prepare) Task=`bfd88fc8-0608-4acd-b25a-7eee8ba89e15`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 13:59:54,433::task::595::Storage.TaskManager.Task::(_updateState) Task=`bfd88fc8-0608-4acd-b25a-7eee8ba89e15`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 13:59:54,433::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 13:59:54,433::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 13:59:54,433::task::993::Storage.TaskManager.Task::(_decref) Task=`bfd88fc8-0608-4acd-b25a-7eee8ba89e15`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 13:59:54,459::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 13:59:54,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:55,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913814 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:57,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:57,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091487 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 13:59:59,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 13:59:59,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913077 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 13:59:59,591::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 13:59:59,603::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 13:59:59,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00522599 s, 71.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 13:59:59,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000623934 s, 838 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:00:00,266::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:00:00,267::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:00:01,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:01,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.49909 s, 2.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:03,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:03,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910382 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:05,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:05,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091601 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:08,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:08,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914996 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:00:09,471::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:00:09,620::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:00:09,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:00:09,643::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00541852 s, 68.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:00:09,644::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652089 s, 802 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:10,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:00:10,279::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:00:10,280::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:00:10,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539852 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:12,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:13,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546295 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:00:14,603::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:00:15,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:15,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915568 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:17,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:17,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913355 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:19,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:19,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912192 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:00:19,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:00:19,659::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:00:19,670::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00374466 s, 99.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:00:19,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00114383 s, 457 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:00:20,289::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:00:20,290::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:00:21,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:21,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912493 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:23,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:23,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914176 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:00:24,483::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:00:25,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:25,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912259 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:27,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:28,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914469 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:00:29,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:00:29,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:00:29,698::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687295 s, 540 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:00:29,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656699 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:30,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:00:30,301::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:00:30,302::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:00:30,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913429 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:32,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:32,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913092 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:34,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:35,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.517517 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:37,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:37,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913922 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:39,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:39,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914661 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:00:39,494::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:00:39,703::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:00:39,715::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:00:39,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000819822 s, 453 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:00:39,733::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000807588 s, 648 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:00:40,309::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:00:40,310::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:00:41,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:42,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.35539 s, 756 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:00:44,635::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:00:44,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:44,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091377 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:46,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:47,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912241 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:49,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:49,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515268 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:00:49,736::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:00:49,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:00:49,763::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000743055 s, 499 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:00:49,764::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713358 s, 733 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:00:50,322::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:00:50,323::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:00:51,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:51,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914391 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:53,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:54,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559498 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:00:54,497::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2567878-5d42-4990-a77d-3d8e8b6b7231`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:00:54,498::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:00:54,500::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:00:54,500::task::1191::Storage.TaskManager.Task::(prepare) Task=`d2567878-5d42-4990-a77d-3d8e8b6b7231`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:00:54,501::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2567878-5d42-4990-a77d-3d8e8b6b7231`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:00:54,501::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:00:54,501::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:00:54,501::task::993::Storage.TaskManager.Task::(_decref) Task=`d2567878-5d42-4990-a77d-3d8e8b6b7231`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:00:54,526::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:00:56,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:56,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097296 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:00:58,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:00:58,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091184 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:00:59,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:00:59,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:00:59,794::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647632 s, 573 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:00:59,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000804286 s, 650 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:01:00,334::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:01:00,335::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:01:00,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:01,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558345 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:03,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:03,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914906 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:05,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:05,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550387 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:07,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:07,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091336 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:01:09,540::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:01:09,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:01:09,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:01:09,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000789343 s, 470 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:01:09,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000957499 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:09,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:10,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091323 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:01:10,341::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:01:10,342::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:01:12,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:12,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913082 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:14,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:14,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915211 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:01:14,667::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:01:16,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:16,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.52067 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:18,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:18,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914628 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:01:19,834::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:01:19,845::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:01:19,857::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000769286 s, 482 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:01:19,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631526 s, 828 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:01:20,349::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:01:20,350::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:01:20,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:21,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556949 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:23,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:23,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913461 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:01:24,553::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:01:25,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:27,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36137 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:29,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:29,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552158 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:01:29,862::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:01:29,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:01:29,884::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667854 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:01:29,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000799166 s, 654 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:01:30,359::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:01:30,360::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:01:31,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:31,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0987201 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:33,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:34,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545285 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:36,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:36,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913775 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:38,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:39,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545089 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:01:39,565::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:01:39,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:01:39,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639204 s, 580 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:01:39,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:01:39,916::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684338 s, 764 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:01:40,371::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:01:40,372::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:01:41,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:41,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914764 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:43,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:43,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.517568 s, 2.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:01:44,701::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:01:45,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:45,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.121288 s, 8.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:47,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:48,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912658 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:01:49,917::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:01:49,929::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:01:49,945::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691942 s, 536 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:01:49,946::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000799223 s, 654 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:50,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:50,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09122 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:01:50,379::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:01:50,380::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:01:52,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:52,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914509 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:54,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:54,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913418 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:01:54,568::task::595::Storage.TaskManager.Task::(_updateState) Task=`6748830c-3c1b-41dc-af65-0b6f1ab083a1`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:01:54,568::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:01:54,571::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:01:54,571::task::1191::Storage.TaskManager.Task::(prepare) Task=`6748830c-3c1b-41dc-af65-0b6f1ab083a1`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:01:54,571::task::595::Storage.TaskManager.Task::(_updateState) Task=`6748830c-3c1b-41dc-af65-0b6f1ab083a1`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:01:54,571::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:01:54,571::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:01:54,571::task::993::Storage.TaskManager.Task::(_decref) Task=`6748830c-3c1b-41dc-af65-0b6f1ab083a1`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:01:54,599::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:01:56,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:56,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912825 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:01:58,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:01:58,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913025 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:01:59,950::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:01:59,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:01:59,962::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000718955 s, 516 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:01:59,975::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662909 s, 789 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:02:00,391::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:02:00,392::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:02:00,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:00,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913391 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:02,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:04,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39933 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:06,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:06,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0931207 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:08,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:08,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913102 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:02:09,612::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 14:02:09,978::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:02:09,982::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:02:09,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000696465 s, 533 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:02:10,010::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:02:10,012::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:02:10,015::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:02:10,016::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:02:10,017::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:02:10,017::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:02:10,018::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:02:10,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000745575 s, 701 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:02:10,400::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:02:10,401::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:02:10,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:10,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09129 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:12,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:12,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914266 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:14,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 14:02:14,734::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:02:15,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.506013 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:17,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:17,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912851 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:19,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:19,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091336 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:02:20,006::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:02:20,019::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654625 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:02:20,034::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:02:20,045::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621051 s, 842 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:02:20,412::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:02:20,413::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:02:21,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:21,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913114 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:23,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:23,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913291 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:02:24,628::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:02:25,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:25,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913131 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:27,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:28,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543416 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:02:30,022::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:02:30,035::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642188 s, 578 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:02:30,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:02:30,060::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000654431 s, 799 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:30,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:02:30,425::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:02:30,426::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:02:30,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914005 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:32,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:32,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914263 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:34,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:34,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912192 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:36,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:38,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37104 s, 747 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:02:39,641::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:02:40,037::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:02:40,050::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:02:40,052::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:02:40,055::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:02:40,056::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:02:40,057::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:02:40,057::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:02:40,058::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:02:40,070::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000640482 s, 579 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:02:40,073::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:02:40,087::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000762859 s, 686 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:40,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:40,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913212 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:02:40,433::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:02:40,434::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:02:42,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:42,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552399 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:02:44,767::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:02:44,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:44,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913337 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:46,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:47,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912685 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:49,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:49,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091548 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:02:50,089::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:02:50,101::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:02:50,112::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000694035 s, 535 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:02:50,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00122772 s, 426 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:02:50,440::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:02:50,441::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:02:51,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:51,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558092 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:53,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:53,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914034 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:02:54,643::task::595::Storage.TaskManager.Task::(_updateState) Task=`32c67fe3-4cc2-481a-ae58-f39f43689ff3`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:02:54,644::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:02:54,647::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:02:54,647::task::1191::Storage.TaskManager.Task::(prepare) Task=`32c67fe3-4cc2-481a-ae58-f39f43689ff3`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:02:54,647::task::595::Storage.TaskManager.Task::(_updateState) Task=`32c67fe3-4cc2-481a-ae58-f39f43689ff3`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:02:54,647::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:02:54,647::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:02:54,648::task::993::Storage.TaskManager.Task::(_decref) Task=`32c67fe3-4cc2-481a-ae58-f39f43689ff3`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:02:54,675::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:02:55,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:56,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913838 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:02:58,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:02:58,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091352 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:03:00,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:03:00,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:03:00,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000829178 s, 447 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:03:00,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000948957 s, 551 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:03:00,452::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:03:00,454::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:03:00,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:00,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914116 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:02,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:02,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910849 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:04,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:05,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.508273 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:07,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:07,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913321 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:09,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:03:09,687::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:03:09,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541339 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:03:10,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:03:10,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000714738 s, 519 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:03:10,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:03:10,178::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686597 s, 762 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:03:10,464::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:03:10,465::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:03:12,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:12,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910287 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:14,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:14,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913113 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:03:14,803::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:03:16,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:16,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100614 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:18,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:18,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924911 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:03:20,179::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:03:20,190::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:03:20,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000742435 s, 500 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:03:20,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000903411 s, 579 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:03:20,471::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:03:20,472::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:03:20,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:20,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914344 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:22,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:22,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910722 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:03:24,700::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:03:24,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:24,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915784 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:26,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:27,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914714 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:29,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:29,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914768 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:03:30,205::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:03:30,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:03:30,227::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000722275 s, 514 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:03:30,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00059647 s, 877 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:03:30,483::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:03:30,485::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:03:31,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:31,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915181 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:33,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:33,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919561 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:35,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:35,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915122 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:37,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:37,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913479 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:39,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:03:39,713::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 14:03:40,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:03:40,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:03:40,249::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000605459 s, 864 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:03:40,264::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684564 s, 542 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:40,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.597581 s, 1.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:03:40,494::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:03:40,495::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:03:42,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:42,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919858 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:44,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:44,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914466 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:03:44,836::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:03:46,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:47,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542304 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:49,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:49,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912814 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:03:50,327::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:03:50,327::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:03:50,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00139199 s, 267 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:03:50,351::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672407 s, 778 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:03:50,506::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:03:50,507::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:03:51,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:51,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913698 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:53,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:53,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912893 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:03:54,716::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b602932-ac9f-4342-8a60-9232da8d538d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:03:54,716::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:03:54,719::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:03:54,719::task::1191::Storage.TaskManager.Task::(prepare) Task=`0b602932-ac9f-4342-8a60-9232da8d538d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:03:54,719::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b602932-ac9f-4342-8a60-9232da8d538d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:03:54,719::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:03:54,719::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:03:54,719::task::993::Storage.TaskManager.Task::(_decref) Task=`0b602932-ac9f-4342-8a60-9232da8d538d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:03:54,744::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:03:55,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:55,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912871 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:57,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:57,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912435 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:03:59,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:03:59,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923644 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:04:00,358::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:04:00,358::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:04:00,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000705773 s, 741 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:04:00,386::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000933643 s, 397 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:04:00,518::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:04:00,519::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:04:01,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:01,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912655 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:03,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:04,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.512188 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:06,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:06,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915134 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:08,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:08,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915918 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:04:09,756::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:04:10,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:04:10,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:04:10,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000602085 s, 616 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:04:10,421::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647294 s, 808 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:04:10,525::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:04:10,526::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:04:10,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:10,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910282 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:12,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:12,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091389 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:04:14,869::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:04:14,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:15,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914933 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:17,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:17,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913131 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:19,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:19,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546726 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:04:20,428::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:04:20,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:04:20,449::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00141049 s, 263 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:04:20,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000792143 s, 660 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:04:20,536::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:04:20,538::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:04:21,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:21,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913783 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:23,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:23,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092171 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:04:24,768::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:04:25,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:26,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545757 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:28,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:28,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091408 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:04:30,455::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:04:30,466::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:04:30,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063674 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:04:30,484::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000760772 s, 687 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:04:30,546::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:04:30,547::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:04:30,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:30,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914149 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:32,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:33,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542757 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:35,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:35,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916704 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:37,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:37,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913475 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:39,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:39,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913533 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:04:39,782::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:04:40,486::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:04:40,498::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:04:40,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00054348 s, 683 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:04:40,515::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693445 s, 754 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:04:40,558::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:04:40,560::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:04:41,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:41,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916612 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:43,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:43,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918156 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:04:44,904::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:04:45,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:46,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.098519 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:48,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:48,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091499 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:50,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:50,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912626 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:04:50,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:04:50,530::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:04:50,541::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00537191 s, 69.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:04:50,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065675 s, 796 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:04:50,571::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:04:50,572::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:04:52,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:52,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913717 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:54,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:54,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914064 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:04:54,785::task::595::Storage.TaskManager.Task::(_updateState) Task=`11862c93-590c-4b53-9612-572e275e35cc`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:04:54,785::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:04:54,788::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:04:54,788::task::1191::Storage.TaskManager.Task::(prepare) Task=`11862c93-590c-4b53-9612-572e275e35cc`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:04:54,788::task::595::Storage.TaskManager.Task::(_updateState) Task=`11862c93-590c-4b53-9612-572e275e35cc`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:04:54,788::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:04:54,788::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:04:54,788::task::993::Storage.TaskManager.Task::(_decref) Task=`11862c93-590c-4b53-9612-572e275e35cc`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:04:54,814::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:04:56,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:56,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912419 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:04:58,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:04:58,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912342 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:05:00,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:05:00,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:05:00,568::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00199262 s, 186 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:05:00,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649346 s, 805 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:05:00,581::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:05:00,582::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:05:00,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:01,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914784 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:03,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:03,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540598 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:05,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:05,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922748 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:07,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:07,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913888 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:09,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:05:09,838::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:05:09,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914309 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:05:10,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:05:10,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:05:10,596::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655784 s, 566 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:05:10,598::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653318 s, 801 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:05:10,607::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:05:10,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:05:11,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:12,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555789 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:14,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 14:05:14,937::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:05:15,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545829 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:17,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:17,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914429 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:19,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:19,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914409 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:05:20,606::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:05:20,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000616268 s, 602 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:05:20,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:05:20,645::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:05:20,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066284 s, 789 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:05:20,651::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:05:21,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:21,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914591 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:23,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:23,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913188 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:05:24,849::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:05:25,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:26,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545847 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:28,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:28,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542217 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:05:30,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:05:30,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630684 s, 588 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:05:30,649::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:05:30,650::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:05:30,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:05:30,666::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685574 s, 763 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:30,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:31,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546272 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:33,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:33,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915387 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:35,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:35,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091285 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:37,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:37,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913827 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:39,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:39,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913764 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:05:39,861::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:05:40,656::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:05:40,657::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:05:40,670::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000763398 s, 486 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:05:40,674::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:05:40,673::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:05:40,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00157908 s, 331 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:41,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:41,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913986 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:43,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:43,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914119 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:05:44,967::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:05:45,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:46,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541733 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:48,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:48,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910199 s, 11.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:05:50,668::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:05:50,669::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:05:50,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:05:50,683::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:05:51,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:05:51,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672004 s, 552 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:05:51,209::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000709995 s, 737 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:51,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.536535 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:53,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:53,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542693 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:05:54,866::task::595::Storage.TaskManager.Task::(_updateState) Task=`debda069-a7b8-4b21-b2c3-21d1982faf11`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:05:54,866::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:05:54,870::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:05:54,870::task::1191::Storage.TaskManager.Task::(prepare) Task=`debda069-a7b8-4b21-b2c3-21d1982faf11`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:05:54,871::task::595::Storage.TaskManager.Task::(_updateState) Task=`debda069-a7b8-4b21-b2c3-21d1982faf11`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:05:54,871::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:05:54,872::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:05:54,873::task::993::Storage.TaskManager.Task::(_decref) Task=`debda069-a7b8-4b21-b2c3-21d1982faf11`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:05:54,901::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:05:55,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:55,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919129 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:05:57,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:05:58,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913197 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:00,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:00,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913662 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:06:00,680::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:06:00,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:06:01,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:06:01,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:06:01,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000773686 s, 480 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:06:01,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00111312 s, 470 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:02,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:02,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.501427 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:04,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:04,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091104 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:06,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:06,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0932291 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:08,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:09,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912927 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:06:09,914::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:06:10,684::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:06:10,685::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:06:11,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:11,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912679 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:06:11,240::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:06:11,253::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000604208 s, 614 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:06:11,254::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:06:11,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000767262 s, 682 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:13,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:13,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918157 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:06:15,000::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:06:15,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:15,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914111 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:17,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:17,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912523 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:19,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:19,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913289 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:06:20,696::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:06:20,698::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:06:21,269::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:06:21,281::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:06:21,292::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654028 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:06:21,294::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000799659 s, 654 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:21,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:21,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915578 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:23,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:23,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914052 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:06:24,928::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:06:25,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:25,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913747 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:27,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:28,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913711 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:30,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:30,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911211 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:06:30,709::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:06:30,710::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:06:31,298::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:06:31,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:06:31,320::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692858 s, 535 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:06:31,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000886383 s, 590 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:32,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:32,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091304 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:34,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:34,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543947 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:36,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:37,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913412 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:39,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:39,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915468 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:06:39,941::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:06:40,720::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:06:40,721::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:06:41,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:41,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915034 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:06:41,326::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:06:41,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:06:41,348::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00470738 s, 78.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:06:41,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000710946 s, 736 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:43,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:43,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912974 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:06:45,035::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:06:45,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:45,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913446 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:47,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:47,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091316 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:49,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:49,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.093627 s, 10.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:06:50,733::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:06:50,734::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:06:51,355::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:06:51,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:06:51,377::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00115528 s, 453 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:06:51,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672521 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:51,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:51,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913431 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:53,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:53,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912445 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:06:54,944::task::595::Storage.TaskManager.Task::(_updateState) Task=`e93a6dec-490d-4c05-b871-af391e5303eb`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:06:54,944::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:06:54,947::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:06:54,947::task::1191::Storage.TaskManager.Task::(prepare) Task=`e93a6dec-490d-4c05-b871-af391e5303eb`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:06:54,947::task::595::Storage.TaskManager.Task::(_updateState) Task=`e93a6dec-490d-4c05-b871-af391e5303eb`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:06:54,947::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:06:54,948::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:06:54,948::task::993::Storage.TaskManager.Task::(_decref) Task=`e93a6dec-490d-4c05-b871-af391e5303eb`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:06:54,975::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:06:55,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:56,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912997 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:06:58,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:06:58,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913109 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:00,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:07:00,745::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:07:00,746::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:07:00,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546249 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:07:01,386::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:07:01,387::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:07:01,417::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000704554 s, 742 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:07:01,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659713 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:02,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:02,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914316 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:04,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:04,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09131 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:06,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:07,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913668 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:09,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:09,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54352 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:07:09,988::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:07:10,754::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:07:10,755::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:07:11,423::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:07:11,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:07:11,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000747352 s, 496 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:07:11,452::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:07:11,453::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:07:11,456::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:07:11,457::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:07:11,457::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:07:11,457::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:07:11,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:07:11,470::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665571 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:11,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:11,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914302 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:13,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:14,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.51589 s, 2.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:07:15,067::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:07:16,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:16,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999431 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:18,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:18,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912125 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:20,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:07:20,767::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:07:20,769::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:07:20,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922622 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:07:21,446::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:07:21,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654864 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:07:21,474::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:07:21,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000634383 s, 824 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:22,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:22,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912137 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:24,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:07:25,002::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:07:25,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913807 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:27,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:27,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913938 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:29,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:29,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912503 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:07:30,779::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:07:30,781::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:07:31,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:07:32,623::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:07:32,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:07:32,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000727443 s, 719 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:07:32,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627535 s, 591 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:32,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39515 s, 734 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:34,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:34,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913844 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:36,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:37,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544883 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:39,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:39,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548452 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:07:40,015::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:07:40,791::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:07:40,792::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:07:41,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:42,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542334 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:07:42,662::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 14:07:42,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:07:42,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680056 s, 769 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:07:42,692::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:07:42,693::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:07:42,696::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:07:42,698::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:07:42,698::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:07:42,698::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:07:42,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:07:42,711::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668176 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:44,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:44,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914832 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:07:45,099::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:07:46,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:46,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912631 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:48,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:48,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913867 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:07:50,805::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:07:50,806::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:07:50,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:50,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914632 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:07:52,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:07:52,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667145 s, 784 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:07:52,714::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:07:52,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641045 s, 579 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:52,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:53,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913886 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:07:55,018::task::595::Storage.TaskManager.Task::(_updateState) Task=`fe434435-178e-4011-8687-f24fd429d52b`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:07:55,018::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:07:55,021::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:07:55,021::task::1191::Storage.TaskManager.Task::(prepare) Task=`fe434435-178e-4011-8687-f24fd429d52b`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:07:55,021::task::595::Storage.TaskManager.Task::(_updateState) Task=`fe434435-178e-4011-8687-f24fd429d52b`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:07:55,022::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:07:55,022::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:07:55,022::task::993::Storage.TaskManager.Task::(_decref) Task=`fe434435-178e-4011-8687-f24fd429d52b`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:07:55,048::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:07:55,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:55,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914217 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:57,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:57,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920201 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:07:59,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:07:59,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913707 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:08:00,817::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:08:00,818::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:08:01,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:01,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913211 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:08:02,703::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:08:02,715::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695981 s, 751 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:08:02,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:08:02,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651439 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:03,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:03,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915213 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:05,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:05,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914898 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:07,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:07,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914534 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:09,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:08:10,061::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:08:10,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55103 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:08:10,829::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:08:10,830::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:08:12,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:12,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913643 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:08:12,719::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:08:12,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635902 s, 822 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:08:12,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:08:12,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066106 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:14,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:14,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911728 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:08:15,133::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:08:16,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:16,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911955 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:18,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:18,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913043 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:08:20,842::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:08:20,843::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:08:20,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:21,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921498 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:08:22,735::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:08:22,748::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000644194 s, 812 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:08:22,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:08:22,770::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622843 s, 596 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:23,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:23,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914887 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:08:25,074::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:08:25,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:25,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913469 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:27,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:27,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913949 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:29,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:29,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091459 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:08:30,854::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:08:30,855::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:08:31,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:31,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913127 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:08:32,751::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:08:32,763::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000708937 s, 738 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:08:32,772::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:08:32,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000606608 s, 612 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:33,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:33,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913582 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:35,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:35,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913775 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:37,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:38,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912737 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:40,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:08:40,090::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:08:40,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912515 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:08:40,863::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:08:40,864::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:08:42,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:42,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914043 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:08:42,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:08:42,782::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660467 s, 792 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:08:42,787::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:08:42,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660834 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:44,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:44,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913568 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:08:45,166::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:08:46,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:46,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.120416 s, 8.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:48,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:49,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912544 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:08:50,876::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:08:50,877::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:08:51,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:51,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914076 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:08:52,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:08:52,798::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686248 s, 762 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:08:52,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:08:52,814::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674738 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:53,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:53,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914473 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:08:55,093::task::595::Storage.TaskManager.Task::(_updateState) Task=`6b2b9a3d-c4a6-44d5-bf9c-14afb3e6a10d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:08:55,093::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:08:55,096::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:08:55,097::task::1191::Storage.TaskManager.Task::(prepare) Task=`6b2b9a3d-c4a6-44d5-bf9c-14afb3e6a10d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:08:55,097::task::595::Storage.TaskManager.Task::(_updateState) Task=`6b2b9a3d-c4a6-44d5-bf9c-14afb3e6a10d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:08:55,097::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:08:55,097::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:08:55,097::task::993::Storage.TaskManager.Task::(_decref) Task=`6b2b9a3d-c4a6-44d5-bf9c-14afb3e6a10d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:08:55,124::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:08:55,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:55,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912513 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:57,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:57,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913874 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:08:59,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:08:59,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912623 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:09:00,889::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:09:00,890::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:09:01,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:01,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913666 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:09:02,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:09:02,814::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006454 s, 810 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:09:02,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:09:02,829::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669102 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:03,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:03,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920557 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:05,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:05,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914462 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:07,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:08,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914952 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:10,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:09:10,137::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:09:10,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913826 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:09:10,898::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:09:10,899::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:09:12,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:12,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913809 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:09:12,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:09:12,829::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683517 s, 765 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:09:12,834::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:09:12,845::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637138 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:14,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:14,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914271 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:09:15,203::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:09:16,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:16,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913469 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:18,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:18,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912628 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:20,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:20,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913388 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:09:20,912::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:09:20,913::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:09:22,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:09:23,343::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:09:23,344::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:09:23,370::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635375 s, 823 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:09:23,374::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648489 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:23,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555845 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:09:25,149::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:09:25,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:25,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924244 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:27,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:27,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912327 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:29,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:29,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913359 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:09:30,924::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:09:30,925::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:09:31,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:31,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914865 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:09:33,382::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:09:33,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648903 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:09:33,398::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:09:33,412::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00198997 s, 263 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:33,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:33,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913072 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:35,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:36,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923635 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:38,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:38,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914842 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:09:40,162::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:09:40,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:40,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092768 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:09:40,935::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:09:40,936::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:09:42,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:42,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558366 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:09:43,412::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:09:43,428::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00059037 s, 628 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:09:43,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:09:43,443::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000625709 s, 836 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:44,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:45,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914573 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:09:45,233::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:09:47,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:47,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914166 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:49,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:49,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515745 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:09:50,947::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:09:50,948::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:09:51,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:51,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912762 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:09:53,444::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:09:53,456::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:09:53,467::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00511508 s, 72.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:09:53,469::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00107463 s, 487 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:53,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:53,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911946 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:09:55,166::task::595::Storage.TaskManager.Task::(_updateState) Task=`eba92dc9-ceb0-4f00-ae37-b0cf83e69880`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:09:55,166::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:09:55,169::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:09:55,169::task::1191::Storage.TaskManager.Task::(prepare) Task=`eba92dc9-ceb0-4f00-ae37-b0cf83e69880`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:09:55,169::task::595::Storage.TaskManager.Task::(_updateState) Task=`eba92dc9-ceb0-4f00-ae37-b0cf83e69880`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:09:55,170::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:09:55,170::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:09:55,170::task::993::Storage.TaskManager.Task::(_decref) Task=`eba92dc9-ceb0-4f00-ae37-b0cf83e69880`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:09:55,197::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:09:55,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:56,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913591 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:09:58,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:09:58,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914707 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:00,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:00,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913659 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:10:00,962::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:10:00,963::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:10:02,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:02,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913926 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:10:03,474::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:10:03,487::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000795052 s, 467 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:10:03,487::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:10:03,501::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000719281 s, 727 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:04,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:04,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914456 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:06,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:06,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913142 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:08,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:08,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913132 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:10:10,209::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:10:10,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:10,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913671 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:10:10,974::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:10:10,976::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:10:12,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:12,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913758 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:10:13,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:10:13,515::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00077037 s, 482 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:10:13,515::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:10:13,528::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000742811 s, 704 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:15,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:15,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916816 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:10:15,265::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:10:17,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:17,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912659 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:19,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:19,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914017 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:10:20,986::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:10:20,987::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:10:21,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:21,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917304 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:23,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:10:23,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:10:23,589::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:10:23,591::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630386 s, 830 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:10:23,605::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676447 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:23,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.130078 s, 7.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:10:25,220::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:10:25,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:25,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913879 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:27,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:27,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914189 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:29,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:29,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913934 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:10:30,999::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:10:31,000::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:10:31,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:32,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914729 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:10:33,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:10:33,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:10:33,641::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000718702 s, 728 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:10:33,643::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000791332 s, 469 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:34,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:34,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551354 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:36,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:37,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.512737 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:39,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:39,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550253 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:10:40,233::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:10:41,010::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:10:41,012::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:10:41,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:41,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913855 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:10:43,650::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:10:43,651::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:10:43,673::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0041086 s, 90.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:10:43,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00063093 s, 829 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:43,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:44,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914991 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:10:45,303::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:10:46,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:46,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100413 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:48,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:48,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091243 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:50,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:50,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555528 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:10:51,018::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:10:51,019::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:10:52,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:53,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913012 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:10:53,680::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:10:53,692::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:10:53,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000760102 s, 488 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:10:53,775::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629228 s, 831 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:55,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:55,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912629 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:10:55,237::task::595::Storage.TaskManager.Task::(_updateState) Task=`b9d05c36-3442-42a3-b134-cafe639e2888`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:10:55,237::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:10:55,240::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:10:55,240::task::1191::Storage.TaskManager.Task::(prepare) Task=`b9d05c36-3442-42a3-b134-cafe639e2888`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:10:55,240::task::595::Storage.TaskManager.Task::(_updateState) Task=`b9d05c36-3442-42a3-b134-cafe639e2888`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:10:55,240::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:10:55,240::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:10:55,241::task::993::Storage.TaskManager.Task::(_decref) Task=`b9d05c36-3442-42a3-b134-cafe639e2888`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:10:55,268::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:10:57,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:10:57,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540269 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:10:59,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:00,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.532161 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:11:01,031::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:11:01,032::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:11:02,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:02,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913851 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:11:03,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:11:03,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:11:03,803::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000793669 s, 467 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:11:03,804::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690473 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:04,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:04,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913191 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:06,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:07,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548973 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:09,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:09,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915211 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:11:10,281::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:11:11,037::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:11:11,039::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:11:11,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:11,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912613 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:13,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:13,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912842 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:11:13,810::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:11:13,810::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:11:13,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645577 s, 810 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:11:13,834::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00121736 s, 305 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:11:15,334::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:11:15,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:15,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916391 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:17,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:17,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921885 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:19,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:19,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091272 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:11:21,050::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:11:21,051::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:11:21,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:22,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916024 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:11:23,839::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:11:23,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:11:23,862::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646924 s, 808 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:11:23,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637081 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:24,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:24,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091271 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:11:25,294::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:11:26,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:26,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913437 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:28,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:28,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0935645 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:30,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:30,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914507 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:11:31,062::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:11:31,064::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:11:32,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:32,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914276 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:11:33,871::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:11:33,871::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:11:33,893::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00211487 s, 247 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:11:33,895::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000852159 s, 435 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:34,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:34,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913266 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:36,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:37,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556688 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:39,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:39,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913647 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:11:40,306::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:11:41,074::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:11:41,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:11:41,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:42,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519085 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:11:43,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:11:43,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:11:43,929::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695926 s, 752 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:11:43,930::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00128834 s, 288 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:44,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:44,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550256 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:11:45,368::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:11:46,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:46,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.114967 s, 8.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:48,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:48,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911711 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:50,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:50,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913432 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:11:51,087::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:11:51,088::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:11:53,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:53,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091551 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:11:53,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:11:53,947::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:11:53,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00183199 s, 285 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:11:53,962::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00193387 s, 192 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:55,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:11:55,310::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7cc463f-21ee-449f-b3a8-d48853e227b3`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:11:55,310::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Dummy-49::DEBUG::2014-07-14 14:11:55,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.333823 s, 3.1 MB/s\n'; <rc> = 0 >Thread-113::INFO::2014-07-14 14:11:55,559::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:11:55,559::task::1191::Storage.TaskManager.Task::(prepare) Task=`c7cc463f-21ee-449f-b3a8-d48853e227b3`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:11:55,559::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7cc463f-21ee-449f-b3a8-d48853e227b3`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:11:55,560::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:11:55,560::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:11:55,560::task::993::Storage.TaskManager.Task::(_decref) Task=`c7cc463f-21ee-449f-b3a8-d48853e227b3`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:11:55,587::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:11:57,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:57,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919053 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:11:59,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:11:59,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912738 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:12:01,099::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:12:01,100::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:12:01,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:01,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913121 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:03,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:03,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914812 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:12:03,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:12:03,967::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:12:03,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00506682 s, 73.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:12:03,989::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000941207 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:05,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:06,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914324 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:08,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:08,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912785 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:10,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:12:10,600::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:12:10,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542576 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:12:11,108::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:12:11,109::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:12:12,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:12,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912776 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:12:13,992::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:12:13,993::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:12:14,012::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000731373 s, 507 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:12:14,027::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:12:14,029::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:12:14,032::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:12:14,033::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:12:14,034::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:12:14,034::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:12:14,035::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:12:14,046::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064056 s, 816 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:14,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:14,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914187 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:12:15,400::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:12:16,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:17,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913633 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:19,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:19,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913032 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:12:21,120::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:12:21,121::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:12:21,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:21,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913001 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:23,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:23,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913791 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:12:24,022::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:12:24,035::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653375 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:12:24,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:12:24,062::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000633318 s, 826 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:12:25,613::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:12:25,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:25,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913749 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:27,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:28,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913333 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:30,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:30,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914296 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:12:31,128::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:12:31,129::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:12:32,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:32,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912938 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:12:34,038::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:12:34,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672182 s, 552 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:12:34,066::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:12:34,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682037 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:34,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:34,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913641 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:36,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:37,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545341 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:39,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:39,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914607 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:12:40,624::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:12:41,138::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:12:41,139::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:12:41,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:41,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914784 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:43,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:43,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912709 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:12:44,052::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:12:44,065::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:12:44,066::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:12:44,069::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:12:44,071::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:12:44,071::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:12:44,071::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:12:44,072::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:12:44,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:12:44,095::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000723542 s, 513 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:12:44,098::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694967 s, 753 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:45,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 14:12:45,438::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:12:45,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914248 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:47,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:47,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911986 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:49,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:49,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919059 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:12:51,150::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:12:51,151::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:12:51,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:51,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914889 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:53,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:12:54,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:12:54,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:12:54,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689403 s, 538 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:12:54,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677983 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:54,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549319 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:12:55,628::task::595::Storage.TaskManager.Task::(_updateState) Task=`6b8b46c3-3dff-4788-8884-28ff095ead60`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:12:55,628::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:12:55,631::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:12:55,631::task::1191::Storage.TaskManager.Task::(prepare) Task=`6b8b46c3-3dff-4788-8884-28ff095ead60`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:12:55,631::task::595::Storage.TaskManager.Task::(_updateState) Task=`6b8b46c3-3dff-4788-8884-28ff095ead60`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:12:55,631::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:12:55,632::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:12:55,632::task::993::Storage.TaskManager.Task::(_decref) Task=`6b8b46c3-3dff-4788-8884-28ff095ead60`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:12:55,659::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:12:56,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:56,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914195 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:12:58,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:12:58,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915688 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:00,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:00,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914543 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:13:01,162::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:13:01,163::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:13:02,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:02,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913118 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:13:04,404::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:13:04,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:13:04,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00122869 s, 302 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:13:04,431::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00126577 s, 413 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:04,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:04,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911987 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:07,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:07,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54532 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:09,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:09,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912405 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:13:10,671::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:13:11,174::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:13:11,175::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:13:11,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:11,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912912 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:13,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:13,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919629 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:13:14,432::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:13:14,445::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:13:14,460::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000768162 s, 483 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:13:14,462::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000753341 s, 694 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:13:15,468::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:13:15,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:16,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967161 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:18,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:18,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913527 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:20,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:20,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912654 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:13:21,186::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:13:21,187::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:13:22,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:22,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559354 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:13:24,465::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:13:24,476::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:13:24,489::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000606133 s, 612 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:13:24,491::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626458 s, 835 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:24,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:25,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912608 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:13:25,683::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:13:27,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:27,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541805 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:29,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:29,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915805 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:13:31,198::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:13:31,200::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:13:31,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:31,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912498 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:33,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:34,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914518 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:13:34,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:13:34,506::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:13:34,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651992 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:13:34,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00152832 s, 342 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:36,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:36,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915505 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:38,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:38,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915747 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:40,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:40,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916807 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:13:40,702::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:13:41,210::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:13:41,211::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:13:42,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:42,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912646 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:44,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:13:44,603::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:13:44,616::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:13:44,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000640131 s, 580 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:13:44,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686612 s, 762 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:44,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.130461 s, 7.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:13:45,496::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:13:46,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:46,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913629 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:48,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:48,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912654 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:50,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:51,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914016 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:13:51,219::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:13:51,220::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:13:53,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:53,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913719 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:13:54,639::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:13:54,650::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:13:54,651::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000744731 s, 498 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:13:54,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648681 s, 806 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:55,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:55,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914322 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:13:55,705::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ee39654-f3c7-4928-938f-c8e172ea280b`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:13:55,705::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:13:55,708::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:13:55,708::task::1191::Storage.TaskManager.Task::(prepare) Task=`4ee39654-f3c7-4928-938f-c8e172ea280b`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:13:55,708::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ee39654-f3c7-4928-938f-c8e172ea280b`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:13:55,708::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:13:55,709::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:13:55,709::task::993::Storage.TaskManager.Task::(_decref) Task=`4ee39654-f3c7-4928-938f-c8e172ea280b`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:13:55,733::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:13:57,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:57,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548465 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:13:59,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:13:59,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912757 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:14:01,230::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:14:01,231::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:14:02,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:02,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557045 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:04,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:14:04,700::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:14:04,702::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:14:04,724::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653074 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:14:04,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696958 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:04,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129093 s, 7.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:06,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:07,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.51994 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:09,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:09,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.133286 s, 7.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:14:10,745::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:14:11,241::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:14:11,242::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:14:11,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:11,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09161 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:13,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:13,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132949 s, 7.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:14:14,733::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:14:14,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:14:14,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0011359 s, 327 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:14:14,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666819 s, 784 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:14:15,528::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:14:15,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:15,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970724 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:17,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:18,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545886 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:20,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:20,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914821 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:14:21,253::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:14:21,255::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:14:22,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:23,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550948 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:14:24,765::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:14:24,776::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:14:24,787::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000608912 s, 609 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:14:24,789::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062157 s, 841 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:25,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:25,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914494 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:14:25,757::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:14:27,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:27,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913061 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:29,471::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:29,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091671 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:14:31,265::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:14:31,266::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:14:31,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:31,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914589 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:33,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:33,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917318 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:14:34,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:14:34,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:14:34,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00284597 s, 130 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:14:34,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000723076 s, 723 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:35,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:35,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916013 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:37,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:38,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911945 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:40,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:40,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091561 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:14:40,769::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:14:41,276::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:14:41,277::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:14:42,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:42,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507073 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:44,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:14:45,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:14:45,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:14:45,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685027 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:14:45,292::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070401 s, 743 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:45,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.574777 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:14:45,558::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:14:47,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:47,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913925 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:49,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:49,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921192 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:14:51,286::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:14:51,287::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:14:51,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:52,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548225 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:54,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:54,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913118 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:14:55,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:14:55,309::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:14:55,311::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685108 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:14:55,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642404 s, 814 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:14:55,772::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f004ab8-d504-4887-bf35-9619e842fbea`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:14:55,772::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:14:55,775::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:14:55,775::task::1191::Storage.TaskManager.Task::(prepare) Task=`0f004ab8-d504-4887-bf35-9619e842fbea`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:14:55,775::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f004ab8-d504-4887-bf35-9619e842fbea`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:14:55,776::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:14:55,776::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:14:55,776::task::993::Storage.TaskManager.Task::(_decref) Task=`0f004ab8-d504-4887-bf35-9619e842fbea`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:14:55,803::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:14:56,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:56,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913209 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:14:58,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:14:58,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.567987 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:01,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:01,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926956 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:15:01,298::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:15:01,299::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:15:03,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:03,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915465 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:05,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:15:05,382::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:15:05,394::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:15:05,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000748982 s, 698 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:15:05,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699929 s, 530 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:05,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.130682 s, 7.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:07,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:07,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916389 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:09,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:09,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918979 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:15:10,815::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:15:11,310::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:15:11,311::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:15:11,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:11,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913461 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:13,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:13,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916678 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:15:15,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:15:15,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:15:15,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00194465 s, 191 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:15:15,447::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720618 s, 726 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:15:15,591::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:15:15,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:16,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0973818 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:18,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:18,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912861 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:20,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:20,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915289 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:15:21,322::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:15:21,324::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:15:22,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:22,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913238 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:24,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:25,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511044 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:15:25,450::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:15:25,467::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641024 s, 579 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:15:25,468::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:15:25,481::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698176 s, 749 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:15:25,828::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:15:27,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:27,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917226 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:29,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:29,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914164 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:31,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:15:31,332::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:15:31,333::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:15:31,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914111 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:33,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:33,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914736 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:15:35,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:15:35,495::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:15:35,506::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658967 s, 563 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:15:35,508::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000797049 s, 656 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:35,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:35,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914494 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:37,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:37,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912679 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:39,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:40,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544552 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:15:40,841::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:15:41,343::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:15:41,344::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:15:42,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:42,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913805 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:44,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:44,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914133 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:15:45,511::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:15:45,523::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:15:45,525::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00059013 s, 629 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:15:45,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679989 s, 769 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:15:45,624::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:15:46,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:46,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913114 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:48,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:49,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.516104 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:51,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:15:51,352::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:15:51,353::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:15:51,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0946103 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:53,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:53,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913846 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:55,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:15:55,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:15:55,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:15:55,635::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000452739 s, 819 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:15:55,636::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000789277 s, 663 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:55,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102865 s, 10.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:15:55,845::task::595::Storage.TaskManager.Task::(_updateState) Task=`6ab6e053-fbc3-4da7-afb6-87f81e22d100`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:15:55,845::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:15:55,848::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:15:55,848::task::1191::Storage.TaskManager.Task::(prepare) Task=`6ab6e053-fbc3-4da7-afb6-87f81e22d100`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:15:55,848::task::595::Storage.TaskManager.Task::(_updateState) Task=`6ab6e053-fbc3-4da7-afb6-87f81e22d100`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:15:55,849::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:15:55,849::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:15:55,849::task::993::Storage.TaskManager.Task::(_decref) Task=`6ab6e053-fbc3-4da7-afb6-87f81e22d100`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:15:55,877::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:15:57,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:57,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916244 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:15:59,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:15:59,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917647 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:16:01,361::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:16:01,362::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:16:01,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:01,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914502 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:04,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:04,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913412 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:16:06,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:16:06,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:16:06,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665334 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:16:06,149::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645032 s, 811 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:06,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:06,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091533 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:08,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:09,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.33543 s, 767 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:16:10,890::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:16:11,371::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:16:11,372::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:16:11,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:12,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545507 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:14,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:14,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541395 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:16:15,657::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 14:16:16,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:16:16,169::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00061155 s, 607 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:16:16,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:16:16,189::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673336 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:16,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:16,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926014 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:18,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:18,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913927 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:21,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:21,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091339 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:16:21,384::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:16:21,386::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:16:23,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:23,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913265 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:25,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:25,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913356 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:16:25,903::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:16:26,179::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:16:26,191::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646729 s, 574 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:16:26,196::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:16:26,208::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666807 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:27,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:27,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912799 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:29,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:29,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913774 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:16:31,393::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:16:31,395::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:16:31,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:31,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914073 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:33,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:33,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914339 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:35,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:35,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912739 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:16:36,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:16:36,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645944 s, 574 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:16:36,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:16:36,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000704675 s, 742 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:37,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:38,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918888 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:40,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:40,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557331 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:16:40,915::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:16:41,405::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:16:41,406::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:16:42,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:42,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914215 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:44,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:45,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541978 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:16:45,691::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 14:16:46,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:16:46,224::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000709517 s, 523 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:16:46,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:16:46,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679385 s, 770 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:47,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:47,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911093 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:49,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:49,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091251 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:16:51,412::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:16:51,413::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:16:51,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:52,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913232 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:54,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:54,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556485 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:16:55,919::task::595::Storage.TaskManager.Task::(_updateState) Task=`8c13329c-aea7-4f9f-a079-4472456e27dd`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:16:55,919::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:16:55,922::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:16:55,922::task::1191::Storage.TaskManager.Task::(prepare) Task=`8c13329c-aea7-4f9f-a079-4472456e27dd`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:16:55,922::task::595::Storage.TaskManager.Task::(_updateState) Task=`8c13329c-aea7-4f9f-a079-4472456e27dd`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:16:55,923::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:16:55,923::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:16:55,923::task::993::Storage.TaskManager.Task::(_decref) Task=`8c13329c-aea7-4f9f-a079-4472456e27dd`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:16:55,950::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:16:56,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:16:56,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674512 s, 550 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:16:56,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:16:56,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683409 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:56,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:56,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912458 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:16:58,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:16:59,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541853 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:01,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:17:01,420::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:17:01,421::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:17:01,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918277 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:03,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:03,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913552 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:05,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:05,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914184 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:17:06,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:17:06,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000631661 s, 587 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:17:06,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:17:06,281::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670792 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:07,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:07,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913581 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:09,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:09,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913742 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:17:10,965::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:17:11,432::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:17:11,433::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:17:11,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:12,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913251 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:14,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:14,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913567 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:17:15,723::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:17:16,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:17:16,284::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:17:17,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:17:17,561::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00133117 s, 279 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:17,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37784 s, 743 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:17:17,588::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:17:17,591::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:17:17,596::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:17:17,601::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:17:17,601::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:17:17,602::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:17:17,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:17:17,617::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000644184 s, 812 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:19,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:20,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558715 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:17:21,444::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:17:21,445::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:17:22,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:22,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914681 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:24,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:24,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913604 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:17:25,978::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:17:26,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:26,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913465 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:17:27,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:17:27,582::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000619411 s, 599 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:17:27,625::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:17:27,637::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669755 s, 781 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:28,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:28,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912706 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:30,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:30,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913225 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:17:31,452::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:17:31,453::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:17:32,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:32,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912698 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:34,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:35,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913697 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:37,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:37,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914444 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:17:37,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:17:37,597::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653002 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:17:37,640::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:17:37,652::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684318 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:39,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:39,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541598 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:17:40,989::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:17:41,462::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:17:41,463::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:17:41,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:41,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920675 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:43,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:44,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541057 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:17:45,756::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:17:46,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:46,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0984257 s, 10.4 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:17:47,599::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:17:47,612::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:17:47,613::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:17:47,616::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:17:47,618::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:17:47,618::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:17:47,618::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:17:47,619::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:17:47,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675652 s, 549 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:17:47,655::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:17:47,667::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650067 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:48,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:48,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914628 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:50,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:50,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912533 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:17:51,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:17:51,475::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:17:52,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:52,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915335 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:54,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:55,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914234 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:17:55,993::task::595::Storage.TaskManager.Task::(_updateState) Task=`ecac7171-754c-4e51-befe-e4c38891a485`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:17:55,993::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:17:55,996::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:17:55,996::task::1191::Storage.TaskManager.Task::(prepare) Task=`ecac7171-754c-4e51-befe-e4c38891a485`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:17:55,996::task::595::Storage.TaskManager.Task::(_updateState) Task=`ecac7171-754c-4e51-befe-e4c38891a485`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:17:55,996::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:17:55,996::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:17:55,997::task::993::Storage.TaskManager.Task::(_decref) Task=`ecac7171-754c-4e51-befe-e4c38891a485`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:17:56,022::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:17:57,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:57,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913023 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:17:57,635::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:17:57,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690756 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:17:57,671::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:17:57,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675941 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:17:59,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:17:59,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912775 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:01,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:01,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912374 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:18:01,487::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:18:01,489::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:18:03,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:03,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914077 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:05,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:05,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091386 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:07,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:18:07,717::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:18:07,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:18:07,739::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693282 s, 754 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:18:07,745::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00181654 s, 204 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:07,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.126265 s, 8.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:09,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:10,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503812 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:18:11,033::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:18:11,498::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:18:11,499::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:18:12,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:12,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550544 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:14,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:14,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912317 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:18:15,790::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:18:16,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:17,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916446 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:18:17,753::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:18:17,764::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:18:17,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000612148 s, 606 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:18:17,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686961 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:19,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:19,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914608 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:21,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:21,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913649 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:18:21,506::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:18:21,507::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:18:23,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:23,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091391 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:25,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:25,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914613 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:18:26,044::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:18:27,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:27,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913168 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:18:27,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:18:27,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:18:27,813::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000700871 s, 529 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:18:27,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662762 s, 789 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:29,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:29,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913381 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:18:31,516::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:18:31,517::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:18:31,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:31,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912705 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:33,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:34,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913175 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:36,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:36,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913866 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:18:37,820::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:18:37,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:18:37,847::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00077904 s, 476 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:18:37,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688029 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:38,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:38,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922405 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:40,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:40,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091326 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:18:41,057::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:18:41,526::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:18:41,527::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:18:42,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:42,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091319 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:44,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:44,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915181 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:18:45,846::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:18:46,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:46,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913102 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:18:47,853::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:18:47,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:18:47,874::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00202514 s, 183 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:18:47,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649961 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:48,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:48,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913519 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:50,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:50,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091278 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:18:51,535::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:18:51,536::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:18:52,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:53,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091392 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:55,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:55,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912717 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:18:56,060::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2e7d33a-ae9d-47ca-9e0e-73298be99c4d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:18:56,061::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:18:56,064::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:18:56,064::task::1191::Storage.TaskManager.Task::(prepare) Task=`b2e7d33a-ae9d-47ca-9e0e-73298be99c4d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:18:56,064::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2e7d33a-ae9d-47ca-9e0e-73298be99c4d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:18:56,064::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:18:56,064::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:18:56,064::task::993::Storage.TaskManager.Task::(_decref) Task=`b2e7d33a-ae9d-47ca-9e0e-73298be99c4d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:18:56,092::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:18:57,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:57,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913414 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:18:57,880::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:18:57,891::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:18:57,906::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679241 s, 546 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:18:57,907::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720384 s, 726 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:18:59,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:18:59,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914377 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:01,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:19:01,546::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:19:01,548::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:19:01,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916367 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:03,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:03,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915226 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:05,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:05,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918388 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:07,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:19:07,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:19:07,924::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:19:07,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000709102 s, 738 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:19:07,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625435 s, 593 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:07,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129078 s, 7.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:09,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:10,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915949 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:19:11,104::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:19:11,558::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:19:11,560::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:19:12,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:12,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914061 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:14,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:14,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915149 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:19:15,878::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:19:16,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:16,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0992137 s, 10.3 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:19:17,950::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:19:17,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:19:17,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00537814 s, 69.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:19:17,973::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000710017 s, 737 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:18,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:19,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50253 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:21,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:21,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914575 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:19:21,571::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:19:21,572::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:19:23,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:23,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911284 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:25,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:25,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915009 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:19:26,117::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:19:27,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:27,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915004 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:19:27,977::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:19:27,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:19:28,003::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000606692 s, 612 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:19:28,005::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000740218 s, 707 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:29,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:29,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914789 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:19:31,583::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:19:31,584::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:19:31,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:31,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914606 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:33,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:33,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911527 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:35,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:36,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913806 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:19:38,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:38,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:19:38,023::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:19:38,024::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636906 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:19:38,130::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000886457 s, 590 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:38,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101879 s, 10.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:40,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:40,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502144 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:19:41,129::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:19:41,595::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:19:41,597::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:19:42,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:42,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913494 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:44,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:44,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913429 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:19:45,920::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:19:46,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:47,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912123 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:19:48,132::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:19:48,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:19:48,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000940154 s, 395 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:19:48,157::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675695 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:49,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:49,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914053 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:51,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:19:51,604::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:19:51,606::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:19:51,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503637 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:53,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:53,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916307 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:19:55,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:55,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913263 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:19:56,133::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f39985e-5984-4a7e-96a7-791e0044ce98`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:19:56,133::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:19:56,136::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:19:56,136::task::1191::Storage.TaskManager.Task::(prepare) Task=`4f39985e-5984-4a7e-96a7-791e0044ce98`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:19:56,136::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f39985e-5984-4a7e-96a7-791e0044ce98`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:19:56,137::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:19:56,137::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:19:56,137::task::993::Storage.TaskManager.Task::(_decref) Task=`4f39985e-5984-4a7e-96a7-791e0044ce98`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:19:56,163::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:19:57,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:19:58,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914596 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:19:58,158::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:19:58,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617784 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:19:58,173::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:19:58,184::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659727 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:00,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:00,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914505 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:20:01,616::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:20:01,617::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:20:02,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:02,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913455 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:04,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:04,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911796 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:06,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:06,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.514499 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:20:08,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:20:08,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704844 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:20:08,189::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:20:08,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626827 s, 834 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:08,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:09,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912717 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:11,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:11,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914894 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:20:11,183::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:20:11,625::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:20:11,626::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:20:13,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:13,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913807 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:15,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:15,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915735 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:20:15,960::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:20:17,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:17,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515419 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:20:18,191::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:20:18,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687638 s, 540 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:20:18,205::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:20:18,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671607 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:19,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:20,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909351 s, 11.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:20:21,637::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:20:21,638::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:20:22,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:22,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913779 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:24,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:24,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924477 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:20:26,200::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:20:26,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:26,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912502 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:20:28,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:20:28,223::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645333 s, 575 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:20:28,224::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:20:28,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064681 s, 809 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:28,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:28,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511228 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:30,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:31,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913441 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:20:31,649::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:20:31,650::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:20:33,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:33,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091227 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:35,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:35,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913696 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:37,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:37,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915869 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:20:38,238::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:20:38,249::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:20:38,260::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065878 s, 563 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:20:38,261::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671337 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:39,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:39,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511226 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:20:41,211::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:20:41,661::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:20:41,662::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:20:41,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:42,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914406 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:44,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:44,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920642 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:20:45,993::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:20:46,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:46,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970754 s, 10.5 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:20:48,264::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:20:48,276::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:20:48,277::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692988 s, 535 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:20:48,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631068 s, 829 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:48,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:48,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914019 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:50,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:50,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091253 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:20:51,674::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:20:51,675::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:20:52,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:52,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915243 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:54,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:54,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913661 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:20:56,214::task::595::Storage.TaskManager.Task::(_updateState) Task=`c1bd1152-6c19-458b-b5cd-928dd372d3ac`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:20:56,214::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:20:56,217::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:20:56,217::task::1191::Storage.TaskManager.Task::(prepare) Task=`c1bd1152-6c19-458b-b5cd-928dd372d3ac`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:20:56,217::task::595::Storage.TaskManager.Task::(_updateState) Task=`c1bd1152-6c19-458b-b5cd-928dd372d3ac`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:20:56,218::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:20:56,218::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:20:56,218::task::993::Storage.TaskManager.Task::(_decref) Task=`c1bd1152-6c19-458b-b5cd-928dd372d3ac`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:20:56,245::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:20:56,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:56,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913463 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:20:58,292::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:20:58,303::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:20:58,304::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000781425 s, 475 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:20:58,317::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00079402 s, 659 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:20:59,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:20:59,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919056 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:01,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:01,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914248 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:21:01,686::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:21:01,687::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:21:03,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:03,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507198 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:05,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:06,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539385 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:21:08,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:21:08,331::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000786838 s, 472 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:21:08,334::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:21:08,348::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000638488 s, 819 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:08,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:08,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913988 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:10,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:11,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559855 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:21:11,257::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:21:11,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:21:11,696::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:21:13,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:13,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912517 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:15,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:15,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914579 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:21:16,032::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:21:17,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:17,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913091 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:21:18,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:21:18,361::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665487 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:21:18,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:21:18,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681814 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:19,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:19,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515963 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:21:21,707::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:21:21,708::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:21:21,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:22,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54237 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:24,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:24,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914218 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:21:26,271::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:21:26,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:26,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918734 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:21:28,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:21:28,378::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621781 s, 597 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:21:28,381::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:21:28,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668131 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:28,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:29,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518696 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:31,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:21:31,712::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:21:31,713::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:21:32,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39901 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:34,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:35,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.520903 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:37,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:37,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.51998 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:21:38,383::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:21:38,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705439 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:21:38,408::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:21:38,419::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653941 s, 800 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:39,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:40,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511071 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:21:41,283::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:21:41,724::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:21:41,726::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:21:42,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:42,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912334 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:44,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:44,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914591 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:21:46,066::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:21:46,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:46,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913252 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:21:48,398::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:21:48,410::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674559 s, 550 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:21:48,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:21:48,435::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663351 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:48,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:49,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.516247 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:51,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:21:51,736::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:21:51,737::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:21:52,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.33745 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:54,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:54,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913876 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:21:56,286::task::595::Storage.TaskManager.Task::(_updateState) Task=`8442b7e4-923c-45eb-ba40-c431eda2a91a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:21:56,286::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:21:56,289::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:21:56,289::task::1191::Storage.TaskManager.Task::(prepare) Task=`8442b7e4-923c-45eb-ba40-c431eda2a91a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:21:56,290::task::595::Storage.TaskManager.Task::(_updateState) Task=`8442b7e4-923c-45eb-ba40-c431eda2a91a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:21:56,290::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:21:56,290::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:21:56,290::task::993::Storage.TaskManager.Task::(_decref) Task=`8442b7e4-923c-45eb-ba40-c431eda2a91a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:21:56,317::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:21:56,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:57,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556518 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:21:58,942::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:21:58,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:21:58,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0053601 s, 69.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:21:58,967::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00063915 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:21:59,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:21:59,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914474 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:01,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:01,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913716 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:22:01,750::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:22:01,751::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:22:03,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:03,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913174 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:05,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:06,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.521333 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:08,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:08,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913571 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:22:08,967::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:22:08,980::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:22:08,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648329 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:22:08,996::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000765166 s, 684 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:10,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:10,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916362 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:22:11,329::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:22:11,763::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:22:11,764::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:22:12,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:13,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542966 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:15,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:15,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915894 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:22:16,100::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:22:17,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:17,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091337 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:22:18,999::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:22:19,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:22:19,016::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000714401 s, 519 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:22:19,030::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:22:19,035::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:22:19,038::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:22:19,039::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:22:19,040::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:22:19,040::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:22:19,041::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:22:19,052::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658927 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:19,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:19,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917239 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:21,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:21,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912705 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:22:21,773::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:22:21,774::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:22:23,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:23,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913518 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:25,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:25,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912597 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:22:26,342::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:22:27,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:27,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091251 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:22:29,035::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:22:29,047::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623558 s, 595 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:22:29,056::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:22:29,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651784 s, 802 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:29,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:29,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912895 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:22:31,781::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:22:31,782::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:22:31,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:32,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918703 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:34,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:34,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542206 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:36,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:36,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913892 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:38,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:38,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913122 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:22:39,051::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:22:39,070::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00605844 s, 61.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:22:39,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:22:39,095::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00457972 s, 114 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:40,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:22:41,355::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:22:41,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558935 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:22:41,788::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:22:41,790::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:22:43,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:44,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550151 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:46,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 14:22:46,133::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:22:46,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922235 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:48,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:22:49,076::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Dummy-49::DEBUG::2014-07-14 14:22:49,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36317 s, 751 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:22:49,706::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:22:49,721::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626157 s, 835 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:22:49,734::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:22:49,735::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:22:49,738::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:22:49,740::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:22:49,740::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:22:49,740::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:22:49,741::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:22:49,753::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654687 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:51,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:51,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913577 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:22:51,801::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:22:51,802::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:22:53,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:53,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912862 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:22:55,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:55,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914241 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:22:56,358::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fe58bc6-a5d4-4d31-b544-7a60deea8214`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:22:56,359::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:22:56,361::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:22:56,361::task::1191::Storage.TaskManager.Task::(prepare) Task=`1fe58bc6-a5d4-4d31-b544-7a60deea8214`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:22:56,361::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fe58bc6-a5d4-4d31-b544-7a60deea8214`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:22:56,362::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:22:56,362::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:22:56,362::task::993::Storage.TaskManager.Task::(_decref) Task=`1fe58bc6-a5d4-4d31-b544-7a60deea8214`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:22:56,389::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:22:57,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:22:58,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091733 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:22:59,732::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:22:59,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642993 s, 813 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:22:59,755::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:22:59,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671833 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:00,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:00,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913362 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:23:01,809::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:23:01,810::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:23:02,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:02,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920881 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:04,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:04,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912858 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:06,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:06,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919167 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:08,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:08,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912537 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:23:09,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:23:09,759::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703307 s, 744 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:23:09,769::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:23:09,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688323 s, 539 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:10,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:10,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913113 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:23:11,402::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:23:11,818::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:23:11,819::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:23:12,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:13,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518708 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:15,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:15,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912906 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:23:16,167::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:23:17,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:18,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54332 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:23:19,763::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:23:19,775::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006701 s, 780 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:23:19,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:23:19,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648319 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:20,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:20,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913065 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:23:21,830::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:23:21,831::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:23:22,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:22,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091302 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:24,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:24,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912845 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:26,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:23:26,418::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:23:26,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912553 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:28,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:28,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.13237 s, 7.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:23:29,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:23:29,791::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694458 s, 753 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:23:29,798::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:23:29,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652617 s, 568 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:30,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:30,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914574 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:23:31,842::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:23:31,843::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:23:32,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:32,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914756 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:34,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:35,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.508126 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:37,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:37,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545113 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:23:39,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:23:39,807::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068084 s, 768 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:23:39,813::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:23:39,825::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647414 s, 573 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:40,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:40,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547563 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:23:41,431::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:23:41,853::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:23:41,854::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:23:42,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:42,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913624 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:44,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:44,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914048 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:23:46,200::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:23:46,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:46,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913893 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:48,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:49,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913246 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:23:49,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:23:49,823::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000628894 s, 832 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:23:49,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:23:49,839::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636614 s, 583 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:51,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:51,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913244 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:23:51,862::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:23:51,863::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:23:53,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:53,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913614 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:55,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:55,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914116 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:23:56,434::task::595::Storage.TaskManager.Task::(_updateState) Task=`a7014ef7-f298-4dea-943b-7dec15cfccae`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:23:56,434::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:23:56,437::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:23:56,437::task::1191::Storage.TaskManager.Task::(prepare) Task=`a7014ef7-f298-4dea-943b-7dec15cfccae`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:23:56,437::task::595::Storage.TaskManager.Task::(_updateState) Task=`a7014ef7-f298-4dea-943b-7dec15cfccae`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:23:56,438::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:23:56,438::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:23:56,438::task::993::Storage.TaskManager.Task::(_decref) Task=`a7014ef7-f298-4dea-943b-7dec15cfccae`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:23:56,465::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:23:57,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:57,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911621 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:23:59,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:23:59,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913016 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:23:59,827::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:23:59,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00131497 s, 398 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:23:59,850::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:23:59,874::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674011 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:01,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:01,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913857 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:24:01,873::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:24:01,875::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:24:03,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:04,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556379 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:06,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:06,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914504 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:08,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:08,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913408 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:24:09,875::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:24:09,887::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668804 s, 782 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:24:09,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:24:09,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000802421 s, 462 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:10,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:10,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913748 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:24:11,478::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:24:11,882::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:24:11,883::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:24:12,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:12,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913157 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:14,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:14,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916846 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:24:16,233::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:24:16,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:17,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911986 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:19,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:19,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091366 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:24:19,905::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:24:19,917::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:24:19,928::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00541132 s, 96.6 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:24:19,930::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632752 s, 586 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:21,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:21,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914289 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:24:21,892::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:24:21,893::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:24:23,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:23,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912464 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:25,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:25,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916511 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:24:26,489::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:24:27,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:27,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915175 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:29,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:29,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922465 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:24:29,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:24:29,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:24:29,963::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000608385 s, 610 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:24:29,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626806 s, 834 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:31,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:31,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091386 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:24:31,900::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:24:31,902::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:24:33,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:34,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542128 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:36,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:36,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915156 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:38,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:38,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913148 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:24:39,969::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:24:39,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:24:39,982::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000739296 s, 502 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:24:39,995::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000732719 s, 714 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:40,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:41,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541807 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:24:41,501::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:24:41,907::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:24:41,908::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:24:43,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:43,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912227 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:45,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:45,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556847 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:24:46,268::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:24:48,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:48,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913991 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:24:49,998::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:24:50,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:24:50,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658931 s, 563 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:24:50,026::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659951 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:50,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:50,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912558 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:24:51,919::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:24:51,920::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:24:52,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:52,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543441 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:54,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:54,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914673 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:24:56,505::task::595::Storage.TaskManager.Task::(_updateState) Task=`30e0fc38-af6e-4245-b303-20aa82356fb9`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:24:56,505::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:24:56,508::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:24:56,508::task::1191::Storage.TaskManager.Task::(prepare) Task=`30e0fc38-af6e-4245-b303-20aa82356fb9`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:24:56,508::task::595::Storage.TaskManager.Task::(_updateState) Task=`30e0fc38-af6e-4245-b303-20aa82356fb9`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:24:56,508::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:24:56,508::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:24:56,509::task::993::Storage.TaskManager.Task::(_decref) Task=`30e0fc38-af6e-4245-b303-20aa82356fb9`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:24:56,534::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:24:56,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:57,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914317 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:24:59,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:24:59,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557456 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:25:00,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:25:00,031::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:25:00,057::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00062404 s, 595 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:25:00,060::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667977 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:01,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:01,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091508 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:25:01,927::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:25:01,928::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:25:03,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:03,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914205 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:05,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:06,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914236 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:08,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:08,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914896 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:25:10,064::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:25:10,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:25:10,092::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629068 s, 590 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:25:10,094::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682153 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:10,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:10,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914596 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:25:11,548::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:25:11,939::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:25:11,940::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:25:12,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:12,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912927 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:14,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:14,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915063 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:25:16,302::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:25:16,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:17,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.530507 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:19,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:19,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911999 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:25:20,097::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:25:20,109::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:25:20,125::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651493 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:25:20,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066159 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:21,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:21,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914192 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:25:21,951::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:25:21,952::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:25:23,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:23,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912459 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:25,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:25,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913933 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:25:26,560::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:25:27,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:27,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091324 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:29,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:30,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091311 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:25:30,130::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:25:30,141::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:25:30,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643373 s, 577 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:25:30,155::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000904005 s, 579 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:25:31,963::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:25:31,964::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:25:32,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:32,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091255 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:34,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:34,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920406 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:36,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:36,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545201 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:38,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:38,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912953 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:25:40,159::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:25:40,171::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:25:40,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00510376 s, 72.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:25:40,185::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667078 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:40,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:41,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542521 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:25:41,572::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:25:41,975::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:25:41,976::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:25:43,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:44,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558467 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:46,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:46,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0992725 s, 10.3 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:25:46,336::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:25:48,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:48,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545626 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:25:50,188::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:25:50,200::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:25:50,202::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674769 s, 550 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:25:50,215::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000757266 s, 691 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:50,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:51,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543675 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:25:51,983::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:25:51,984::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:25:53,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:53,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913541 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:55,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:55,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913385 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:25:56,575::task::595::Storage.TaskManager.Task::(_updateState) Task=`8adcfd1d-7359-4f56-96f1-89a82b09d46f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:25:56,575::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:25:56,578::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:25:56,578::task::1191::Storage.TaskManager.Task::(prepare) Task=`8adcfd1d-7359-4f56-96f1-89a82b09d46f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:25:56,578::task::595::Storage.TaskManager.Task::(_updateState) Task=`8adcfd1d-7359-4f56-96f1-89a82b09d46f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:25:56,579::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:25:56,579::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:25:56,579::task::993::Storage.TaskManager.Task::(_decref) Task=`8adcfd1d-7359-4f56-96f1-89a82b09d46f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:25:56,604::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:25:57,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:25:57,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914393 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:25:59,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:26:00,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:26:00,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:26:00,389::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00345889 s, 151 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:26:00,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662291 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:00,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.529169 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:26:01,996::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:26:01,997::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:26:02,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:02,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912923 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:04,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:05,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559698 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:07,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:07,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545275 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:09,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:10,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914253 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:26:10,402::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:26:10,414::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657646 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:26:10,416::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:26:10,429::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000704044 s, 743 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:26:11,617::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:26:12,008::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:26:12,009::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:26:12,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:12,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915847 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:14,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:14,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913494 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:16,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:16,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100582 s, 10.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:26:16,369::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:26:18,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:18,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912613 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:26:20,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:26:20,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:26:20,442::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644738 s, 575 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:26:20,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000807625 s, 648 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:20,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:20,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911968 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:26:22,020::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:26:22,021::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:26:22,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:22,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913613 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:24,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:24,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912127 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:26:26,630::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:26:26,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:27,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:29,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:29,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:26:30,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:26:30,469::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:26:30,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705313 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:26:30,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643301 s, 813 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:31,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:31,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912226 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:26:32,030::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:26:32,031::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:26:33,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:34,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519928 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:36,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:36,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914235 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:38,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:38,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913228 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:26:40,484::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:26:40,498::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000990454 s, 375 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:26:40,501::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:26:40,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000830274 s, 630 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:40,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:40,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913233 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:26:41,642::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:26:42,042::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:26:42,043::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:26:42,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:42,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912906 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:44,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:44,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915272 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:26:46,403::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:26:46,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:47,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540073 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:49,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:49,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920612 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:26:50,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:26:50,531::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:26:50,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704667 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:26:50,549::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687124 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:51,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:51,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912225 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:26:52,055::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:26:52,056::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:26:53,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:53,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913429 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:26:55,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:55,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913686 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:26:56,645::task::595::Storage.TaskManager.Task::(_updateState) Task=`5eb537ef-5766-4b28-aecf-0b41c99c212a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:26:56,646::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:26:56,649::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:26:56,649::task::1191::Storage.TaskManager.Task::(prepare) Task=`5eb537ef-5766-4b28-aecf-0b41c99c212a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:26:56,649::task::595::Storage.TaskManager.Task::(_updateState) Task=`5eb537ef-5766-4b28-aecf-0b41c99c212a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:26:56,649::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:26:56,649::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:26:56,649::task::993::Storage.TaskManager.Task::(_decref) Task=`5eb537ef-5766-4b28-aecf-0b41c99c212a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:26:56,677::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:26:57,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:26:58,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914028 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:00,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:27:00,641::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:27:00,656::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000633276 s, 826 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:27:00,659::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:27:00,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000740192 s, 501 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:00,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.587251 s, 1.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:27:02,070::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:27:02,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:27:02,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:02,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913269 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:04,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:05,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549303 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:07,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:07,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914228 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:09,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:09,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911933 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:27:10,684::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:27:10,695::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687262 s, 761 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:27:10,696::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:27:10,710::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00073359 s, 506 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:11,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:27:11,696::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:27:11,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912821 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:27:12,075::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:27:12,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:27:13,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:13,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913207 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:15,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:15,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0975742 s, 10.5 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:27:16,436::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:27:18,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:18,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912481 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:20,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:20,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911907 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:27:20,710::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:27:20,716::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:27:20,732::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000789891 s, 470 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:27:20,744::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:27:20,745::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:27:20,748::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:27:20,749::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:27:20,749::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:27:20,749::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:27:20,750::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:27:20,762::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671358 s, 779 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:27:22,087::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:27:22,088::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:27:22,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:22,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917991 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:24,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:24,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913785 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:26,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:26,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916065 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:27:26,709::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:27:28,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:28,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914042 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:27:30,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:27:30,755::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643938 s, 576 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:27:30,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:27:30,777::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669716 s, 781 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:30,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:30,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913286 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:27:32,099::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:27:32,100::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:27:32,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:33,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916503 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:35,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:35,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911701 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:37,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:37,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091282 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:39,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:39,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502104 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:27:40,759::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:27:40,770::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636152 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:27:40,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:27:40,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684952 s, 764 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:27:41,720::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:27:41,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:41,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912327 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:27:42,110::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:27:42,111::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:27:43,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:44,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914831 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:46,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:46,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100723 s, 10.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:27:46,469::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:27:48,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:48,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926453 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:50,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:50,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914391 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:27:50,772::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:27:50,787::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:27:50,788::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:27:50,791::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:27:50,792::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:27:50,793::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:27:50,793::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:27:50,794::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:27:50,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:27:50,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00458261 s, 81.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:27:50,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695596 s, 752 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:27:52,117::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:27:52,119::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:27:52,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:52,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912813 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:54,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:54,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912683 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:27:56,723::task::595::Storage.TaskManager.Task::(_updateState) Task=`57619ef8-6749-4e55-92be-2c0b2bd7c9d2`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:27:56,724::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:27:56,726::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:27:56,726::task::1191::Storage.TaskManager.Task::(prepare) Task=`57619ef8-6749-4e55-92be-2c0b2bd7c9d2`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:27:56,726::task::595::Storage.TaskManager.Task::(_updateState) Task=`57619ef8-6749-4e55-92be-2c0b2bd7c9d2`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:27:56,727::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:27:56,727::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:27:56,727::task::993::Storage.TaskManager.Task::(_decref) Task=`57619ef8-6749-4e55-92be-2c0b2bd7c9d2`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:27:56,752::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:27:56,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:56,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915268 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:27:58,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:27:59,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542797 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:28:00,822::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:28:00,834::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:28:00,836::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000955557 s, 388 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:28:00,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637631 s, 820 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:01,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:02,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559005 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:28:02,129::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:28:02,130::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:28:04,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:04,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:06,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:06,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543597 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:08,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:09,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546397 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:28:10,853::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:28:10,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:28:10,875::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663888 s, 559 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:28:10,877::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000726564 s, 720 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:11,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:11,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913378 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:28:11,764::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:28:12,141::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:28:12,143::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:28:13,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:13,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911695 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:15,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:15,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100212 s, 10.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:28:16,504::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:28:17,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:17,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913293 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:19,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:20,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091299 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:28:20,881::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:28:20,892::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:28:20,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636329 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:28:20,905::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658474 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:22,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:22,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911895 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:28:22,151::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:28:22,153::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:28:24,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:24,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913069 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:26,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:28:26,776::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:28:26,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504315 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:28,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:28,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912668 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:28:30,911::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:28:30,922::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:28:30,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625618 s, 593 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:30,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:28:30,940::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639559 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:31,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542114 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:28:32,164::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:28:32,165::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:28:33,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:33,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912191 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:35,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:35,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913829 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:37,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:38,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502487 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:40,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:40,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913079 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:28:40,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:28:40,968::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699039 s, 531 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:28:41,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:28:41,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000705341 s, 741 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:28:41,788::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:28:42,176::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:28:42,177::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:28:42,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:42,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100803 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:44,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:44,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914458 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:28:46,542::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:28:46,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:46,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913556 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:48,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:49,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913404 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:28:50,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:28:50,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672323 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:51,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:51,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914425 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:28:51,510::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:28:51,524::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00265643 s, 197 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:28:52,188::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:28:52,189::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:28:53,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:53,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542204 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:28:55,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:28:56,791::task::595::Storage.TaskManager.Task::(_updateState) Task=`117299fd-18b7-4123-9b4a-0b1fb02b485a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:28:56,792::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:28:57,092::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:28:57,093::task::1191::Storage.TaskManager.Task::(prepare) Task=`117299fd-18b7-4123-9b4a-0b1fb02b485a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:28:57,093::task::595::Storage.TaskManager.Task::(_updateState) Task=`117299fd-18b7-4123-9b4a-0b1fb02b485a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:28:57,094::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:28:57,095::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:28:57,095::task::993::Storage.TaskManager.Task::(_decref) Task=`117299fd-18b7-4123-9b4a-0b1fb02b485a`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 14:28:57,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.369 s, 748 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:28:57,124::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:28:59,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:28:59,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510844 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:29:00,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:29:00,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675866 s, 549 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:29:01,528::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:29:01,539::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070734 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:01,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:01,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912783 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:29:02,200::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:29:02,201::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:29:03,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:03,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913553 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:05,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:05,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915453 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:08,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:08,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915174 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:10,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:10,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913464 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:29:11,002::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:29:11,014::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678113 s, 547 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:29:11,543::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:29:11,554::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692022 s, 756 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:29:12,136::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:29:12,210::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:29:12,211::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:29:12,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:12,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916366 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:14,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:14,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914993 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:16,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 14:29:16,576::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:29:16,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0929501 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:18,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:18,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092111 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:20,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:20,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913563 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:29:21,018::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:29:21,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674868 s, 550 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:29:21,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:29:21,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00072455 s, 722 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:29:22,222::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:29:22,223::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:29:22,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:22,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913574 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:24,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:25,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913423 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:27,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:29:27,148::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:29:27,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543159 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:29,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:29,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091217 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:29:31,033::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:29:31,046::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000613842 s, 604 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:29:31,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:29:31,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661367 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:31,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:31,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914594 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:29:32,234::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:29:32,236::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:29:33,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:34,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.516861 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:36,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:36,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910445 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:38,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:38,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913738 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:40,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:29:41,397::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:29:41,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651087 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:41,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559939 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:29:41,589::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:29:41,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656275 s, 797 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:29:42,160::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 14:29:42,247::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:29:42,248::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:29:43,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:43,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914863 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:45,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:46,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549281 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:29:46,609::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:29:48,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:48,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914201 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:50,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:50,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540672 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:29:51,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:29:51,429::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676657 s, 548 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:29:51,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:29:51,616::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669671 s, 781 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:29:52,259::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:29:52,260::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:29:52,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:52,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0982339 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:55,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:55,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548844 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:29:57,163::task::595::Storage.TaskManager.Task::(_updateState) Task=`7abe054e-9ad6-49e2-a372-876b36167a23`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:29:57,163::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Dummy-49::DEBUG::2014-07-14 14:29:57,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::INFO::2014-07-14 14:29:57,671::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:29:57,671::task::1191::Storage.TaskManager.Task::(prepare) Task=`7abe054e-9ad6-49e2-a372-876b36167a23`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:29:57,671::task::595::Storage.TaskManager.Task::(_updateState) Task=`7abe054e-9ad6-49e2-a372-876b36167a23`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:29:57,671::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:29:57,672::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:29:57,672::task::993::Storage.TaskManager.Task::(_decref) Task=`7abe054e-9ad6-49e2-a372-876b36167a23`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:29:57,698::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:29:57,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913221 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:29:59,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:29:59,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913822 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:30:01,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:30:01,444::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677312 s, 548 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:30:01,620::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:30:01,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006553 s, 798 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:01,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:01,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926502 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:30:02,271::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:30:02,272::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:30:04,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:04,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923344 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:06,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:06,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.505184 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:08,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:09,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546287 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:11,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:11,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925673 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:30:11,447::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:30:11,459::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673842 s, 551 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:30:11,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:30:11,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668071 s, 783 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:30:12,283::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:30:12,284::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:30:12,710::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:30:13,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:13,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915155 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:15,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:16,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.553018 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:30:16,650::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:30:18,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:18,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092015 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:20,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:20,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54385 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:30:21,462::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:30:21,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065371 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:30:21,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:30:21,660::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652512 s, 802 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:30:22,295::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:30:22,296::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:30:22,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:24,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.33589 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:26,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:26,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914852 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:30:27,721::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:30:28,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:28,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914661 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:30,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:30,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913319 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:30:31,477::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:30:31,489::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690179 s, 538 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:30:31,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:30:31,677::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000654007 s, 800 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:30:32,307::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:30:32,308::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:30:32,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:32,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913242 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:34,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:34,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913453 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:36,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:36,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917058 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:38,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:39,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910874 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:41,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:41,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917981 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:30:41,492::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:30:41,504::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069094 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:30:41,681::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:30:41,693::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703379 s, 744 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:30:42,312::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:30:42,313::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:30:42,732::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:30:43,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:43,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912649 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:45,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:45,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921242 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:30:46,696::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:30:47,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:47,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913416 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:49,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:49,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914258 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:30:51,507::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:30:51,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630931 s, 588 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:51,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:30:51,729::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:30:51,742::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661081 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:51,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10806 s, 9.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:30:52,320::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:30:52,322::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:30:53,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:53,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913507 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:30:55,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:55,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913882 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:30:57,741::task::595::Storage.TaskManager.Task::(_updateState) Task=`228ca10e-ec7d-47c3-b1ed-761fc88d82d0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:30:57,742::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:30:57,745::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:30:57,745::task::1191::Storage.TaskManager.Task::(prepare) Task=`228ca10e-ec7d-47c3-b1ed-761fc88d82d0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:30:57,745::task::595::Storage.TaskManager.Task::(_updateState) Task=`228ca10e-ec7d-47c3-b1ed-761fc88d82d0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:30:57,745::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:30:57,745::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:30:57,746::task::993::Storage.TaskManager.Task::(_decref) Task=`228ca10e-ec7d-47c3-b1ed-761fc88d82d0`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:30:57,772::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:30:57,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:30:58,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091279 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:00,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:00,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0935453 s, 10.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:31:01,522::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:31:01,537::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000756769 s, 490 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:31:01,751::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:31:01,763::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650215 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:02,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:31:02,332::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:31:02,334::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:31:02,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543292 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:04,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:04,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915614 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:06,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:07,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.385058 s, 2.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:09,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:09,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091353 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:11,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:31:11,553::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:31:11,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000631752 s, 587 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:11,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10845 s, 9.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:31:11,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:31:11,778::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661728 s, 790 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:31:12,345::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:31:12,346::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:31:12,787::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:31:13,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:13,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914548 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:15,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:15,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0976619 s, 10.5 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:31:16,726::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:31:17,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:18,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913488 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:20,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:20,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912817 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:31:21,572::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:31:21,583::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693241 s, 535 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:31:21,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:31:21,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649423 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:22,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:22,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091271 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:31:22,357::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:31:22,358::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:31:24,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:24,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912896 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:26,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:26,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913147 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:31:27,801::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:31:28,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:28,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912103 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:30,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:30,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914586 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:31:31,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:31:31,599::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632774 s, 586 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:31:31,796::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:31:31,809::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641484 s, 815 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:31:32,369::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:31:32,370::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:31:32,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:32,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912926 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:34,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:34,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914348 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:36,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:37,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510217 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:39,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:39,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911662 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:31:41,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:31:41,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653165 s, 568 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:41,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:41,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916711 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:31:41,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:31:41,825::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661054 s, 791 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:31:42,382::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:31:42,383::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:31:42,818::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:31:43,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:44,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547303 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:46,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:46,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100333 s, 10.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:31:46,760::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:31:48,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:48,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.112658 s, 9.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:50,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:50,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913194 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:31:51,617::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:31:51,628::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683773 s, 543 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:31:51,829::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:31:51,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000625883 s, 836 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:31:52,394::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:31:52,395::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:31:52,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:52,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0959367 s, 10.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:54,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:55,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091173 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:31:57,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:57,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913791 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:31:57,821::task::595::Storage.TaskManager.Task::(_updateState) Task=`76d62db2-cf33-4344-92c2-068d8cbbf647`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:31:57,821::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:31:57,825::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:31:57,825::task::1191::Storage.TaskManager.Task::(prepare) Task=`76d62db2-cf33-4344-92c2-068d8cbbf647`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:31:57,825::task::595::Storage.TaskManager.Task::(_updateState) Task=`76d62db2-cf33-4344-92c2-068d8cbbf647`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:31:57,825::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:31:57,825::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:31:57,825::task::993::Storage.TaskManager.Task::(_decref) Task=`76d62db2-cf33-4344-92c2-068d8cbbf647`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:31:57,850::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:31:59,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:31:59,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913558 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:01,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:32:01,833::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:32:01,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000993984 s, 373 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:32:01,849::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:01,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.538746 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:32:01,868::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00576462 s, 90.7 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:32:02,407::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:32:02,408::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:32:03,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:03,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913996 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:06,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:06,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914743 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:08,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:08,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:10,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:10,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091354 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:32:11,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:32:11,865::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.001508 s, 246 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:32:11,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:32:11,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666882 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:12,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:32:12,416::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:32:12,417::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:32:12,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913099 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:32:12,872::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:32:14,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:14,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913438 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:16,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:16,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911999 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:32:16,793::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:32:18,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:18,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911926 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:20,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:20,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912798 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:32:21,868::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:32:21,880::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691253 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:32:21,890::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:32:21,907::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:32:21,908::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:32:21,911::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:32:21,913::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:32:21,913::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:32:21,913::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:32:21,914::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:32:21,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655478 s, 798 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:32:22,427::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:32:22,428::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:32:22,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:23,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541486 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:25,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:25,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913233 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:27,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:27,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912928 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:32:27,889::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:32:29,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:30,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912463 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:32:31,884::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:32:31,896::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000612076 s, 606 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:32:31,929::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:32:31,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000644251 s, 812 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:32,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:32,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912186 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:32:32,439::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:32:32,440::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:32:34,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:34,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912042 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:36,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:36,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913458 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:38,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:38,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912803 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:40,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:40,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913351 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:32:41,899::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:32:41,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623601 s, 595 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:32:41,944::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:32:41,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678303 s, 771 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:32:42,449::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:32:42,450::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:32:42,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:32:42,915::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:32:43,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919431 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:45,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:45,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914146 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:32:46,828::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:32:47,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:47,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909653 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:49,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:49,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914606 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:51,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:32:51,913::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Dummy-49::DEBUG::2014-07-14 14:32:51,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559525 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:32:51,955::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:32:51,958::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:32:51,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:32:51,972::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-29::DEBUG::2014-07-14 14:32:51,975::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00161799 s, 323 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:32:51,981::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:32:51,984::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:32:51,985::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:32:51,990::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:32:52,004::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000609814 s, 608 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:32:52,457::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:32:52,458::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:32:54,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:54,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918585 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:32:56,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:56,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914197 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:32:57,919::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d2faa3b-da33-4da1-930f-f57ea6018741`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:32:57,919::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:32:57,922::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:32:57,922::task::1191::Storage.TaskManager.Task::(prepare) Task=`6d2faa3b-da33-4da1-930f-f57ea6018741`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:32:57,922::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d2faa3b-da33-4da1-930f-f57ea6018741`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:32:57,922::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:32:57,922::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:32:57,922::task::993::Storage.TaskManager.Task::(_decref) Task=`6d2faa3b-da33-4da1-930f-f57ea6018741`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:32:57,949::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:32:58,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:32:58,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914862 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:00,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:00,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912724 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:33:01,990::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:33:02,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662928 s, 789 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:33:02,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:33:02,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671526 s, 552 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:33:02,469::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:33:02,471::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:33:02,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:03,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.506046 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:05,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:05,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916397 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:07,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:07,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914393 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:09,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:09,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912556 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:11,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:11,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913818 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:33:12,005::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:33:12,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679529 s, 770 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:33:12,024::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:33:12,036::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671373 s, 553 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:33:12,475::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:33:12,476::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:33:12,963::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:33:13,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:14,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914851 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:16,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:16,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100208 s, 10.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:33:16,860::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:33:18,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:18,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091314 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:20,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:20,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912473 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:33:22,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:33:22,032::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667156 s, 784 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:33:22,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:33:22,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000602758 s, 616 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:33:22,488::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:33:22,489::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:33:22,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:22,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912728 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:24,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:25,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557053 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:27,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:27,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092139 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:33:27,976::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:33:29,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:29,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091256 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:31,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:31,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913607 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:33:32,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:33:32,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668877 s, 782 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:33:32,053::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:33:32,066::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641688 s, 578 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:33:32,500::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:33:32,501::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:33:33,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:33,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914556 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:35,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:35,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915397 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:37,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:37,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091252 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:39,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:40,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916033 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:33:42,052::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:33:42,064::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682651 s, 766 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:33:42,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:33:42,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653625 s, 568 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:42,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:33:42,508::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:33:42,509::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:33:42,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510438 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:33:42,994::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:33:44,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:45,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554891 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:33:46,895::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:33:47,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:47,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915479 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:49,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:49,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913266 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:51,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:51,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913198 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:33:52,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:33:52,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067066 s, 780 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:33:52,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:33:52,095::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669977 s, 554 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:33:52,520::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:33:52,521::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:33:53,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:53,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913799 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:55,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:33:55,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912447 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:33:57,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:33:57,998::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3da8b44-9b3f-4d0e-946e-ba624de3fd83`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:33:57,998::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:33:58,345::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:33:58,345::task::1191::Storage.TaskManager.Task::(prepare) Task=`b3da8b44-9b3f-4d0e-946e-ba624de3fd83`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:33:58,346::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3da8b44-9b3f-4d0e-946e-ba624de3fd83`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:33:58,346::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:33:58,346::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:33:58,347::task::993::Storage.TaskManager.Task::(_decref) Task=`b3da8b44-9b3f-4d0e-946e-ba624de3fd83`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 14:33:58,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544671 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:33:58,378::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:34:00,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:00,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911628 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:34:02,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:34:02,096::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691265 s, 757 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:34:02,100::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:34:02,112::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671433 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:02,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:34:02,533::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:34:02,534::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:34:02,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915571 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:04,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:04,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913326 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:06,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:07,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910895 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:09,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:09,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912336 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:11,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:11,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912657 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:34:12,101::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:34:12,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694567 s, 753 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:34:12,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:34:12,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661634 s, 561 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:34:12,542::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:34:12,544::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:34:13,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:34:13,391::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:34:13,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913705 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:15,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:15,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914979 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:34:16,928::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:34:17,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:17,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913182 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:19,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:19,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911683 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:21,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:21,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914011 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:34:22,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:34:22,143::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:34:22,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00129039 s, 405 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:34:22,155::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000711857 s, 521 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:34:22,560::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:34:22,561::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:34:23,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:24,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.535501 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:26,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:26,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091255 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:34:28,404::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:34:28,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:28,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913073 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:30,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:30,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0973309 s, 10.5 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:34:32,161::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:34:32,172::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:34:32,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667772 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:34:32,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00063971 s, 818 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:34:32,570::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:34:32,571::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:34:32,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:32,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913246 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:34,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:35,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549006 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:37,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:37,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911124 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:39,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:39,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914772 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:41,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:41,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912986 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:34:42,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:34:42,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:34:42,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626488 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:34:42,215::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637045 s, 821 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:34:42,580::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:34:42,582::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:34:43,416::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:34:43,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:43,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919539 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:45,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:46,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0983714 s, 10.4 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:34:46,961::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:34:48,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:48,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912546 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:50,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:50,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912183 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:34:52,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:34:52,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:34:52,241::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00509861 s, 72.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:34:52,242::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00073143 s, 715 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:52,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:34:52,593::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:34:52,594::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:34:53,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558253 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:55,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:55,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913769 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:34:57,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:57,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913251 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:34:58,419::task::595::Storage.TaskManager.Task::(_updateState) Task=`534995f2-b1a4-4883-812f-b874ec087b51`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:34:58,419::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:34:58,422::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:34:58,422::task::1191::Storage.TaskManager.Task::(prepare) Task=`534995f2-b1a4-4883-812f-b874ec087b51`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:34:58,422::task::595::Storage.TaskManager.Task::(_updateState) Task=`534995f2-b1a4-4883-812f-b874ec087b51`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:34:58,422::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:34:58,423::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:34:58,423::task::993::Storage.TaskManager.Task::(_decref) Task=`534995f2-b1a4-4883-812f-b874ec087b51`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:34:58,448::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:34:59,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:34:59,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091205 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:01,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:01,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50855 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:35:02,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:35:02,258::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:35:02,275::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000631354 s, 588 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:35:02,276::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066008 s, 792 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:35:02,600::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:35:02,601::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:35:03,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:04,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091162 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:06,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:06,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919169 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:08,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:08,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913875 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:10,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:10,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915397 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:35:12,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:35:12,293::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:35:12,304::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000709211 s, 737 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:35:12,305::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000855134 s, 434 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:12,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:12,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913665 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:35:12,608::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:35:12,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:35:13,460::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:35:14,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:14,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913448 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:16,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:16,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912112 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:35:16,993::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:35:18,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:18,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913362 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:20,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:20,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091209 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:35:22,314::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:35:22,314::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:35:22,339::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679165 s, 770 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:35:22,343::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000751867 s, 493 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:35:22,616::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:35:22,617::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:35:22,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:23,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912313 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:25,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:25,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914364 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:27,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:27,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911455 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:35:28,473::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:35:29,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:29,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910761 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:31,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:31,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938536 s, 10.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:35:32,346::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:35:32,357::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:35:32,368::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000719601 s, 727 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:35:32,371::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000928603 s, 400 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:35:32,627::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:35:32,628::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:35:33,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:33,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914097 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:35,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:35,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914012 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:37,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:37,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911937 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:39,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:40,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914151 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:42,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:42,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913989 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:35:42,374::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:35:42,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:35:42,401::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000620389 s, 843 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:35:42,403::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000732317 s, 507 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:35:42,636::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:35:42,637::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:35:43,484::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:35:44,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:44,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914348 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:46,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:46,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100213 s, 10.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:35:47,025::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:35:48,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:48,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913853 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:50,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:50,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911876 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:35:52,409::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:35:52,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:35:52,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646581 s, 574 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:35:52,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641072 s, 816 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:35:52,648::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:35:52,649::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:35:52,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:52,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912479 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:54,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:54,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092057 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:35:56,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:57,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911385 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:35:58,488::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a30a11e-89ee-4ce4-ac97-525402b8065c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:35:58,488::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:35:58,491::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:35:58,492::task::1191::Storage.TaskManager.Task::(prepare) Task=`5a30a11e-89ee-4ce4-ac97-525402b8065c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:35:58,492::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a30a11e-89ee-4ce4-ac97-525402b8065c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:35:58,492::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:35:58,492::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:35:58,492::task::993::Storage.TaskManager.Task::(_decref) Task=`5a30a11e-89ee-4ce4-ac97-525402b8065c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:35:58,520::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:35:59,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:35:59,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913113 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:01,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:01,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911877 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:36:02,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:36:02,447::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000725139 s, 512 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:36:02,448::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:36:02,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000627986 s, 833 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:36:02,660::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:36:02,661::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:36:03,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:04,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507882 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:06,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:06,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914865 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:08,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:08,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914119 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:10,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:11,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913473 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:36:12,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:36:12,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000720508 s, 515 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:36:12,475::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:36:12,489::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631799 s, 828 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:36:12,670::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:36:12,671::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:36:13,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:13,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914215 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:36:13,532::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:36:15,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:15,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914638 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:36:17,055::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:36:17,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:17,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913221 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:19,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:19,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544036 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:21,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:22,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913013 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:36:22,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:36:22,503::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000624353 s, 594 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:36:22,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:36:22,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658444 s, 794 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:36:22,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:36:22,683::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:36:24,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:24,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914042 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:26,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:26,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914125 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:28,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:28,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912405 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:36:28,545::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:36:30,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:30,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914459 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:36:32,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:36:32,530::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:36:32,541::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641158 s, 579 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:36:32,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670121 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:32,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:36:32,691::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:36:32,693::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:36:33,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539893 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:35,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:35,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913143 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:37,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:37,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915089 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:39,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:39,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915414 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:41,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:41,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912395 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:36:42,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:36:42,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:36:42,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000744715 s, 498 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:36:42,570::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641628 s, 815 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:36:42,701::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:36:42,703::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:36:43,566::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:36:43,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:43,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912741 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:45,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:45,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915713 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:36:47,087::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:36:47,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:48,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913795 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:50,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:50,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914662 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:52,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:52,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091387 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:36:52,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:36:52,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:36:52,595::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00612074 s, 60.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:36:52,597::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000628479 s, 832 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:36:52,714::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:36:52,715::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:36:54,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:54,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913732 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:56,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:36:56,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913031 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:36:58,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:36:58,569::task::595::Storage.TaskManager.Task::(_updateState) Task=`408d691d-81f4-4b6d-825e-c44bd90acfed`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:36:58,569::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:36:58,587::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:36:58,587::task::1191::Storage.TaskManager.Task::(prepare) Task=`408d691d-81f4-4b6d-825e-c44bd90acfed`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:36:58,588::task::595::Storage.TaskManager.Task::(_updateState) Task=`408d691d-81f4-4b6d-825e-c44bd90acfed`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:36:58,589::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:36:58,589::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:36:58,590::task::993::Storage.TaskManager.Task::(_decref) Task=`408d691d-81f4-4b6d-825e-c44bd90acfed`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 14:36:58,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.095845 s, 10.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:36:58,618::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:37:00,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:01,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511523 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:37:02,601::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:37:02,612::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:37:02,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652584 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:37:02,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000801593 s, 652 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:37:02,722::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:37:02,723::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:37:03,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:03,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558924 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:05,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:06,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547125 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:08,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:08,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912436 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:10,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:10,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919556 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:12,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:37:12,651::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:37:12,652::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:12,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.117444 s, 8.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:37:12,731::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:37:12,732::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:37:12,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000519985 s, 713 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:37:12,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00149745 s, 349 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:37:13,640::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:37:14,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:14,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912655 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:16,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:16,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913488 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:37:17,118::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:37:18,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:19,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914795 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:21,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:21,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915681 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:37:22,740::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:37:22,741::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:37:22,743::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:37:22,745::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:37:22,765::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000701773 s, 529 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:37:22,777::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:37:22,778::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:37:22,781::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:37:22,783::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:37:22,783::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:37:22,783::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:37:22,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:37:22,796::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667108 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:23,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:23,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914782 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:25,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:25,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914927 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:27,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:27,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916007 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:37:28,651::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:37:29,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:29,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914482 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:31,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:31,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914173 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:37:32,752::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:37:32,753::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:37:32,773::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:37:32,785::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656392 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:37:32,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:37:32,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631336 s, 828 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:33,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:33,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913482 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:35,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:36,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914019 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:38,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:38,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914907 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:40,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:40,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548131 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:42,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:37:42,761::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:37:42,762::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:37:42,845::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:37:42,856::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:37:42,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00276513 s, 189 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:37:42,874::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644939 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:42,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.126666 s, 8.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:37:43,665::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:37:44,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:45,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916065 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:47,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:47,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914441 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:37:47,150::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:37:49,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:49,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.11274 s, 9.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:51,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:51,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912732 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:37:52,768::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:37:52,770::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:37:52,880::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 14:37:52,883::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:37:52,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00184347 s, 284 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:37:52,919::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:37:52,920::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:37:52,923::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:37:52,925::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:37:52,925::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:37:52,925::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:37:52,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:37:52,939::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660233 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:53,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:53,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912776 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:55,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:55,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091396 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:37:57,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:57,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915069 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:37:58,668::task::595::Storage.TaskManager.Task::(_updateState) Task=`71b01067-1a05-4bd3-ba0e-476bbbcf2a99`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:37:58,668::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:37:58,671::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:37:58,671::task::1191::Storage.TaskManager.Task::(prepare) Task=`71b01067-1a05-4bd3-ba0e-476bbbcf2a99`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:37:58,671::task::595::Storage.TaskManager.Task::(_updateState) Task=`71b01067-1a05-4bd3-ba0e-476bbbcf2a99`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:37:58,671::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:37:58,672::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:37:58,672::task::993::Storage.TaskManager.Task::(_decref) Task=`71b01067-1a05-4bd3-ba0e-476bbbcf2a99`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:37:58,699::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:37:59,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:37:59,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912919 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:01,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:01,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913833 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:38:02,780::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:38:02,781::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:38:02,919::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:38:02,930::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635634 s, 823 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:38:02,942::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:38:02,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648723 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:03,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:04,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55832 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:06,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:06,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915087 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:08,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:08,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913321 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:10,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:10,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913723 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:38:12,792::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:38:12,793::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:38:12,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:38:13,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:38:13,015::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:38:13,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646419 s, 809 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:38:13,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635955 s, 583 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:13,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132489 s, 7.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:38:13,711::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:38:15,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:15,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548772 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:38:17,182::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:38:17,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:17,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912148 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:19,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:20,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542933 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:22,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:38:22,804::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:38:22,805::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:38:22,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.535432 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:38:23,051::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:38:23,063::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000638307 s, 581 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:38:23,066::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:38:23,078::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000728408 s, 718 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:25,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:25,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916179 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:27,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:27,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912669 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:38:28,723::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:38:29,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:29,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917514 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:31,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:31,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914943 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:38:32,811::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:38:32,812::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:38:33,079::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:38:33,099::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00614285 s, 60.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:38:33,101::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:38:33,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00571157 s, 91.6 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:33,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:33,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913954 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:35,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:36,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515665 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:38,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:38,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914538 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:40,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:40,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914103 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:42,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:42,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912411 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:38:42,819::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:38:42,821::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:38:43,115::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:38:43,128::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000720416 s, 515 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:38:43,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:38:43,142::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646834 s, 809 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:38:43,735::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:38:44,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:44,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912557 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:46,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:46,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913437 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:38:47,215::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:38:48,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:48,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913465 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:50,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:50,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913458 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:38:52,830::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:38:52,831::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:38:52,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:53,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920325 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:38:53,143::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:38:53,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669171 s, 554 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:38:53,157::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:38:53,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000624479 s, 837 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:55,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:55,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910727 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:38:57,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:57,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132516 s, 7.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:38:58,738::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3b7f1b6-bf16-447b-b449-04bac90932d5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:38:58,738::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:38:58,742::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:38:58,742::task::1191::Storage.TaskManager.Task::(prepare) Task=`b3b7f1b6-bf16-447b-b449-04bac90932d5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:38:58,742::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3b7f1b6-bf16-447b-b449-04bac90932d5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:38:58,742::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:38:58,742::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:38:58,742::task::993::Storage.TaskManager.Task::(_decref) Task=`b3b7f1b6-bf16-447b-b449-04bac90932d5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:38:58,769::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:38:59,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:38:59,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541926 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:01,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:02,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910916 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:39:02,842::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:39:02,844::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:39:03,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:39:03,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:39:03,193::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000736847 s, 503 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:39:03,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720934 s, 725 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:04,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:04,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916033 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:06,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:06,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091151 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:08,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:08,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922887 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:10,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:10,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912916 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:12,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:12,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913328 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:39:12,853::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:39:12,855::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:39:13,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:39:13,209::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:39:13,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000710802 s, 522 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:39:13,227::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000633155 s, 826 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:39:13,782::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:39:14,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:14,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915299 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:16,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:16,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913798 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:39:17,248::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:39:18,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:18,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912738 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:21,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:21,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091331 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:39:22,862::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:39:22,863::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:39:23,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:23,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914429 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:39:23,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:39:23,234::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:39:23,257::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00203929 s, 256 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:39:23,258::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000615908 s, 602 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:25,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:25,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511251 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:27,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:27,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913327 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:39:28,795::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:39:29,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:30,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913514 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:32,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:32,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091302 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:39:32,874::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:39:32,875::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:39:33,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:39:33,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:39:33,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00484124 s, 76.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:39:33,292::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648543 s, 806 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:34,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:34,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917221 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:36,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:36,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914489 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:38,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:38,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544686 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:40,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:41,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540104 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:39:42,883::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:39:42,885::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:39:43,296::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:39:43,307::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:39:43,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000732462 s, 507 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:39:43,324::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681309 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:43,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:39:43,808::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:39:44,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542722 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:46,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:46,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.12677 s, 8.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:39:47,281::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:39:48,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:48,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912088 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:50,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:50,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915115 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:52,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:39:52,896::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:39:52,897::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:39:53,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5572 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:39:53,328::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:39:53,339::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:39:53,350::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000727767 s, 510 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:39:53,353::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000634985 s, 824 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:55,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:55,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910209 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:39:57,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:57,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914074 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:39:58,811::task::595::Storage.TaskManager.Task::(_updateState) Task=`3128d68b-acdd-40ba-bef6-5c7b89ebe44b`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:39:58,812::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:39:58,815::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:39:58,815::task::1191::Storage.TaskManager.Task::(prepare) Task=`3128d68b-acdd-40ba-bef6-5c7b89ebe44b`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:39:58,815::task::595::Storage.TaskManager.Task::(_updateState) Task=`3128d68b-acdd-40ba-bef6-5c7b89ebe44b`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:39:58,815::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:39:58,815::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:39:58,815::task::993::Storage.TaskManager.Task::(_decref) Task=`3128d68b-acdd-40ba-bef6-5c7b89ebe44b`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:39:58,843::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:39:59,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:39:59,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912247 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:01,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:02,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543697 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:40:02,908::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:40:02,909::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:40:03,354::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:40:03,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000731027 s, 508 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:40:03,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:40:03,379::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00080864 s, 647 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:04,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:04,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544572 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:06,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:06,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913718 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:08,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:09,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557814 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:11,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:12,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37065 s, 747 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:40:12,917::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:40:12,918::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:40:13,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:40:13,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:40:13,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00079372 s, 467 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:40:13,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000603144 s, 867 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:40:13,855::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:40:14,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:14,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917553 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:16,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:17,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913202 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:40:17,313::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:40:19,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:19,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545668 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:21,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:21,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912731 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:40:22,925::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:40:22,926::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:40:23,410::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:40:23,422::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:40:23,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00542038 s, 68.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:40:23,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000810466 s, 645 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:23,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:23,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920798 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:25,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:25,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912153 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:27,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:28,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913879 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:40:28,867::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:40:30,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:30,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914715 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:32,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:32,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.509465 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:40:32,935::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:40:32,936::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:40:33,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:40:33,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:40:33,467::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00205962 s, 180 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:40:33,470::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713187 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:34,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:34,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913283 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:36,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:36,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913993 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:38,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:39,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913307 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:41,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:41,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.512023 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:40:42,944::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:40:42,946::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:40:43,470::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:40:43,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:40:43,484::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000602695 s, 616 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:40:43,497::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672605 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:43,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:43,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913941 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:40:43,880::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:40:45,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:46,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552322 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:40:47,347::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:40:48,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:48,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548421 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:51,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:51,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912272 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:40:52,954::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:40:52,955::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:40:53,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:53,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913321 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:40:53,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:40:53,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:40:53,514::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682176 s, 544 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:40:53,527::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000852765 s, 613 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:55,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:55,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915321 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:40:57,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:57,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913823 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:40:58,883::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0c40958-71b9-4d03-88e1-af85aa655a55`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:40:58,883::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:40:58,886::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:40:58,886::task::1191::Storage.TaskManager.Task::(prepare) Task=`c0c40958-71b9-4d03-88e1-af85aa655a55`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:40:58,886::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0c40958-71b9-4d03-88e1-af85aa655a55`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:40:58,887::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:40:58,887::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:40:58,887::task::993::Storage.TaskManager.Task::(_decref) Task=`c0c40958-71b9-4d03-88e1-af85aa655a55`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:40:58,912::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:40:59,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:40:59,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091346 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:01,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:01,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914012 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:41:02,966::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:41:02,967::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:41:03,532::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:41:03,543::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:41:03,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000749569 s, 495 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:41:03,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000633724 s, 825 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:03,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:03,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913215 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:05,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:05,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914595 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:07,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:08,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920552 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:10,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:10,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914961 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:12,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:12,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915598 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:41:12,975::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:41:12,977::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:41:13,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:41:13,572::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000797708 s, 465 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:41:13,574::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:41:13,592::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000896449 s, 583 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:41:13,924::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:41:14,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:14,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913701 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:16,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:16,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0957493 s, 10.7 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:41:17,381::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:41:18,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:18,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914436 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:20,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:20,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913492 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:22,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:22,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912046 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:41:22,988::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:41:22,989::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:41:23,593::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:41:23,606::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:41:23,621::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00153682 s, 241 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:41:23,624::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632054 s, 827 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:24,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:25,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554507 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:27,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:27,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091607 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:41:28,936::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:41:29,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:29,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912363 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:31,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:31,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913528 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:41:33,000::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:41:33,001::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:41:33,629::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:41:33,640::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:41:33,651::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623478 s, 595 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:41:33,653::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00107097 s, 488 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:33,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:34,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55852 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:36,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:36,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540852 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:38,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:39,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913795 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:41,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:41,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915036 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:41:43,012::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:41:43,013::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:41:43,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:41:43,749::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:41:43,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:41:43,773::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663099 s, 789 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:41:43,777::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000750567 s, 494 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:43,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555216 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:41:43,948::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:41:45,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:45,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100434 s, 10.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:41:47,416::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:41:48,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:48,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914074 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:50,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:50,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912564 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:52,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:52,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913236 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:41:53,021::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:41:53,022::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:41:53,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:41:53,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:41:53,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00187058 s, 198 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:41:53,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646091 s, 809 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:54,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:54,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091384 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:41:56,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:57,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.492918 s, 2.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:41:58,950::task::595::Storage.TaskManager.Task::(_updateState) Task=`ae4f8834-5e4d-4a96-8641-cdca4a909d87`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:41:58,951::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:41:58,954::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:41:58,954::task::1191::Storage.TaskManager.Task::(prepare) Task=`ae4f8834-5e4d-4a96-8641-cdca4a909d87`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:41:58,954::task::595::Storage.TaskManager.Task::(_updateState) Task=`ae4f8834-5e4d-4a96-8641-cdca4a909d87`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:41:58,955::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:41:58,955::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:41:58,955::task::993::Storage.TaskManager.Task::(_decref) Task=`ae4f8834-5e4d-4a96-8641-cdca4a909d87`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:41:58,980::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:41:59,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:41:59,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913255 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:01,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:01,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544218 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:42:03,028::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:42:03,030::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:42:03,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:42:03,830::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:42:03,830::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:42:03,855::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645568 s, 575 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:42:03,859::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692784 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:03,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132085 s, 7.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:05,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:05,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914618 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:08,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:08,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914655 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:10,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:10,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916216 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:12,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:12,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914128 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:42:13,038::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:42:13,039::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:42:13,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:42:13,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:42:13,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00610782 s, 60.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:42:13,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064463 s, 811 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:42:13,992::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:42:14,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:14,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091464 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:16,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:16,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.105575 s, 9.7 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:42:17,450::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:42:18,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:18,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913712 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:20,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:20,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914128 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:22,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:42:23,048::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:42:23,049::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:42:23,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911492 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:42:23,894::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:42:23,895::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:42:23,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667294 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:42:23,927::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:42:23,928::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:42:23,931::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:42:23,932::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:42:23,932::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:42:23,932::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:42:23,933::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:42:23,946::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00166611 s, 314 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:25,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:25,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914159 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:27,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:27,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914413 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:42:29,003::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:42:29,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:29,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913709 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:31,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:31,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914035 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:42:33,060::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:42:33,061::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:42:33,924::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:33,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:42:33,945::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00405819 s, 91.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:42:34,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:42:34,051::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000739178 s, 708 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:34,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104351 s, 9.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:36,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:36,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091565 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:38,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:38,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913658 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:40,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:40,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913828 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:42,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:42,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091316 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:42:43,072::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:42:43,073::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:42:43,950::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:42:43,962::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650774 s, 570 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:42:44,015::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 14:42:44,062::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:42:44,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663546 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:44,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:44,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914693 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:46,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:46,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0971235 s, 10.5 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:42:47,484::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:42:48,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:49,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547526 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:51,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:51,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920776 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:42:53,084::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:42:53,085::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:42:53,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:53,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913499 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:42:53,964::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:42:53,983::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:42:53,985::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:42:53,988::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:42:53,989::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:42:53,990::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:42:53,990::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:42:53,991::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:42:54,002::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000778947 s, 476 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:42:54,078::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:42:54,089::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716486 s, 730 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:55,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:55,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914671 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:42:57,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:42:57,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913814 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:42:59,018::task::595::Storage.TaskManager.Task::(_updateState) Task=`43a61224-1997-4255-9e8f-77724b030953`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:42:59,018::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:42:59,022::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:42:59,022::task::1191::Storage.TaskManager.Task::(prepare) Task=`43a61224-1997-4255-9e8f-77724b030953`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:42:59,022::task::595::Storage.TaskManager.Task::(_updateState) Task=`43a61224-1997-4255-9e8f-77724b030953`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:42:59,022::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:42:59,022::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:42:59,022::task::993::Storage.TaskManager.Task::(_decref) Task=`43a61224-1997-4255-9e8f-77724b030953`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:42:59,047::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:42:59,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:00,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913882 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:02,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:02,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914641 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:43:03,094::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:43:03,095::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:43:04,006::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:43:04,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678491 s, 547 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:43:04,093::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:43:04,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00205715 s, 254 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:04,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:04,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915292 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:06,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:06,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09159 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:08,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:08,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913535 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:10,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:10,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912339 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:12,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:12,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091637 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:43:13,106::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:43:13,107::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:43:14,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:43:14,033::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689794 s, 538 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:43:14,059::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 14:43:14,111::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:43:14,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068608 s, 762 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:14,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:15,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912296 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:17,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:17,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091196 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:43:17,525::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:43:19,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:19,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920763 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:21,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:21,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914351 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:43:23,118::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:43:23,119::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:43:23,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:43:24,820::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:43:24,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:43:24,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627117 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:43:24,853::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694082 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:24,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.40787 s, 727 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:26,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:26,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912807 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:28,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:43:29,072::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:43:29,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915561 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:31,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:31,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913914 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:43:33,130::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:43:33,131::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:43:33,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:33,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914323 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:43:34,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:43:34,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:43:34,884::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677433 s, 548 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:43:34,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672152 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:35,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:35,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914115 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:37,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:37,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916685 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:39,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:39,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091322 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:41,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:41,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911429 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:43:43,138::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:43:43,140::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:43:43,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:43,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914404 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:43:44,084::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:43:44,889::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:43:44,900::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:43:44,911::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068152 s, 544 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:43:44,912::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684365 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:45,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:46,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101613 s, 10.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:43:47,558::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:43:48,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:48,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914334 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:50,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:50,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914394 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:52,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:43:53,150::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:43:53,151::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:43:53,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558536 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:43:54,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:43:54,929::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636049 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:43:54,931::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:43:54,943::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658328 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:55,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:55,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542905 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:43:57,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:43:57,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909869 s, 11.3 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:43:59,090::task::595::Storage.TaskManager.Task::(_updateState) Task=`93ffd13a-a6dd-4c24-9f77-2ce820fd9bd5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:43:59,090::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:43:59,093::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:43:59,093::task::1191::Storage.TaskManager.Task::(prepare) Task=`93ffd13a-a6dd-4c24-9f77-2ce820fd9bd5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:43:59,094::task::595::Storage.TaskManager.Task::(_updateState) Task=`93ffd13a-a6dd-4c24-9f77-2ce820fd9bd5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:43:59,094::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:43:59,094::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:43:59,094::task::993::Storage.TaskManager.Task::(_decref) Task=`93ffd13a-a6dd-4c24-9f77-2ce820fd9bd5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:43:59,119::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:43:59,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:00,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091219 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:02,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:02,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914699 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:44:03,162::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:44:03,164::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:44:04,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:04,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914269 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:44:04,944::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:44:04,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:44:04,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000805795 s, 460 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:44:04,969::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00110843 s, 472 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:06,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:06,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915487 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:08,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:08,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091445 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:10,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:10,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912453 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:12,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:12,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914038 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:44:13,172::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:44:13,173::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:44:14,130::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:44:14,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:14,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914444 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:44:14,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:44:14,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:44:14,993::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000813395 s, 456 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:44:14,997::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00101085 s, 517 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:16,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:17,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091409 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:44:17,590::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:44:19,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:19,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917395 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:21,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:21,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551099 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:44:23,184::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:44:23,185::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:44:23,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:23,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914933 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:44:25,000::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:44:25,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:44:25,022::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00607749 s, 61.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:44:25,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000758962 s, 689 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:25,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:25,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915022 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:27,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:28,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914208 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:44:29,142::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:44:30,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:30,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914377 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:32,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:32,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913471 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:44:33,196::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:44:33,197::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:44:34,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:34,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914339 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:44:35,026::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:44:35,038::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:44:35,054::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689026 s, 538 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:44:35,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658456 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:36,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:36,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914749 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:38,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:39,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547481 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:41,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:41,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913953 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:44:43,208::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:44:43,209::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:44:43,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:43,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539834 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:44:44,154::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:44:45,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:44:45,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:44:45,073::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000594647 s, 624 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:44:45,086::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068922 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:45,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:45,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913173 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:44:47,622::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:44:48,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:48,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912734 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:50,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:50,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09126 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:52,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:52,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912896 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:44:53,220::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:44:53,221::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:44:54,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:54,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911082 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:44:55,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:44:55,103::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:44:55,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00545134 s, 68.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:44:55,115::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679318 s, 770 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:56,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:56,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912288 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:44:58,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:44:58,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091364 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:44:59,157::task::595::Storage.TaskManager.Task::(_updateState) Task=`993ebdaf-3271-42a1-baf7-71da42586070`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:44:59,157::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:44:59,160::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:44:59,160::task::1191::Storage.TaskManager.Task::(prepare) Task=`993ebdaf-3271-42a1-baf7-71da42586070`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:44:59,160::task::595::Storage.TaskManager.Task::(_updateState) Task=`993ebdaf-3271-42a1-baf7-71da42586070`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:44:59,161::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:44:59,161::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:44:59,161::task::993::Storage.TaskManager.Task::(_decref) Task=`993ebdaf-3271-42a1-baf7-71da42586070`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:44:59,188::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:45:00,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:00,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914562 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:02,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:02,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913263 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:45:03,231::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:45:03,232::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:45:04,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:05,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914851 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:45:05,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:45:05,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:45:05,143::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648386 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:45:05,145::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639598 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:07,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:07,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914269 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:09,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:09,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910508 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:11,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:11,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912887 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:45:13,241::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:45:13,242::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:45:13,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:13,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912956 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:45:14,201::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:45:15,149::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:45:15,160::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:45:15,171::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691926 s, 536 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:45:15,173::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678082 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:15,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:16,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.535727 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:45:17,661::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:45:18,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:18,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913275 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:20,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:20,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912269 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:22,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:22,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912388 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:45:23,253::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:45:23,254::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:45:24,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:24,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912397 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:45:25,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:45:25,188::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:45:25,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00588816 s, 63.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:45:25,200::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684248 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:26,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:28,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39813 s, 732 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:45:29,214::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:45:30,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:30,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542881 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:32,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:32,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912931 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:45:33,263::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:45:33,264::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:45:34,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:34,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913564 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:45:35,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:45:35,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:45:35,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000607579 s, 611 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:45:35,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690308 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:36,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:36,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912757 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:38,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:39,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938785 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:41,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:41,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091463 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:43,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:45:43,276::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:45:43,277::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:45:43,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911296 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:45:44,227::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:45:45,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:45:45,246::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:45:45,258::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658001 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:45:45,260::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000783208 s, 668 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:45,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:45,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914286 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:47,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:47,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913936 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:45:47,692::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:45:49,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:50,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546119 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:52,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:52,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918717 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:45:53,287::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:45:53,289::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:45:54,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:54,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913017 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:45:55,264::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:45:55,275::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:45:55,286::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644097 s, 576 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:45:55,287::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000846575 s, 618 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:56,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:57,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549362 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:45:59,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:45:59,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914303 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:45:59,230::task::595::Storage.TaskManager.Task::(_updateState) Task=`30daabd4-a153-4013-bed5-bbd02d6b733e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:45:59,230::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:45:59,233::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:45:59,233::task::1191::Storage.TaskManager.Task::(prepare) Task=`30daabd4-a153-4013-bed5-bbd02d6b733e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:45:59,233::task::595::Storage.TaskManager.Task::(_updateState) Task=`30daabd4-a153-4013-bed5-bbd02d6b733e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:45:59,233::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:45:59,234::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:45:59,234::task::993::Storage.TaskManager.Task::(_decref) Task=`30daabd4-a153-4013-bed5-bbd02d6b733e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:45:59,261::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:46:01,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:01,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913042 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:46:03,299::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:46:03,300::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:46:03,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:03,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918211 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:46:05,291::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:46:05,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:46:05,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00284931 s, 130 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:46:05,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00100022 s, 523 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:05,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:05,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912454 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:07,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:08,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544257 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:10,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:10,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544595 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:12,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:12,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911612 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:46:13,311::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:46:13,312::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:46:14,274::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:46:14,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:46:15,339::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:46:15,339::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:46:15,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00303061 s, 173 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:46:15,367::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000868851 s, 427 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:15,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558155 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:17,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:17,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970946 s, 10.5 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:46:17,723::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:46:19,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:19,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913069 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:21,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:21,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913226 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:46:23,323::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:46:23,324::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:46:23,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:23,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915048 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:46:25,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:46:25,387::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:46:25,398::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000744149 s, 499 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:46:25,399::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000724576 s, 722 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:25,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:26,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:28,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:46:29,286::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:46:29,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38102 s, 741 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:31,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:32,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541928 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:46:33,333::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:46:33,334::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:46:34,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:34,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913483 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:46:35,403::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:46:35,415::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:46:35,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00507171 s, 73.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:46:35,427::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000606987 s, 862 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:36,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:36,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914144 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:38,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:38,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915332 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:40,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:40,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09141 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:42,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:42,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913278 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:46:43,342::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:46:43,344::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:46:44,298::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:46:45,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:45,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915352 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:46:45,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:46:45,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:46:45,452::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00436806 s, 84.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:46:45,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000610013 s, 857 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:47,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:47,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920537 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:46:47,757::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:46:49,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:49,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912762 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:51,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:51,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913651 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:46:53,355::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:46:53,356::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:46:53,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:54,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540432 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:46:55,457::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:46:55,468::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:46:55,479::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0025194 s, 147 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:46:55,481::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657641 s, 795 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:56,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:56,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914321 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:46:58,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:46:58,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544275 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:46:59,301::task::595::Storage.TaskManager.Task::(_updateState) Task=`1da6f27b-b041-4b57-9c10-75d39c123624`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:46:59,301::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:46:59,305::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:46:59,306::task::1191::Storage.TaskManager.Task::(prepare) Task=`1da6f27b-b041-4b57-9c10-75d39c123624`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:46:59,306::task::595::Storage.TaskManager.Task::(_updateState) Task=`1da6f27b-b041-4b57-9c10-75d39c123624`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:46:59,306::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:46:59,306::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:46:59,306::task::993::Storage.TaskManager.Task::(_decref) Task=`1da6f27b-b041-4b57-9c10-75d39c123624`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:46:59,333::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:47:00,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:01,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541418 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:03,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:47:03,364::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:47:03,365::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:47:03,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913883 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:05,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:47:05,511::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:47:05,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:47:05,536::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669716 s, 554 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:47:05,540::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000723581 s, 723 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:05,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.128431 s, 8.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:07,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:07,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912886 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:09,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:09,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916774 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:11,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:11,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912819 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:47:13,369::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:47:13,370::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:47:13,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:14,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915599 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:47:14,345::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:47:15,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:47:15,561::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000779409 s, 476 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:47:15,565::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:47:15,583::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00505069 s, 104 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:16,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:16,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540414 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:47:17,790::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:47:18,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:18,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091117 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:20,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:21,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543732 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:23,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:47:23,381::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:47:23,382::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:47:23,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913496 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:25,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:25,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911741 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:47:25,567::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:47:25,579::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000715523 s, 519 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:47:25,589::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:47:25,606::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:47:25,607::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:47:25,610::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:47:25,611::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:47:25,611::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:47:25,611::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:47:25,612::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:47:25,624::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681935 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:27,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:27,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913958 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:47:29,357::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:47:29,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:29,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914178 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:31,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:33,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36244 s, 752 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:47:33,391::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:47:33,392::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:47:35,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:47:35,736::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:47:35,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:47:35,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00479171 s, 77.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:47:35,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629462 s, 831 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:35,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.580898 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:37,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:37,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911666 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:39,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:40,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915669 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:42,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:42,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916771 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:47:43,405::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:47:43,410::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:47:44,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:44,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914317 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:47:44,370::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 14:47:45,765::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:47:45,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000774164 s, 479 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:47:45,783::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:47:45,798::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635839 s, 823 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:46,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:46,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504381 s, 2.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:47:47,823::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:47:48,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:49,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543141 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:51,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:51,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914047 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:47:53,416::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:47:53,417::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:47:53,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:53,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912615 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:55,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:47:55,786::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 14:47:56,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:47:56,157::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00079516 s, 658 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:56,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.572293 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:47:56,194::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:47:56,196::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:47:56,200::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:47:56,204::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:47:56,205::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:47:56,206::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:47:56,208::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:47:56,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000747247 s, 496 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:47:58,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:47:58,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912602 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:47:59,373::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4d19fb1-9bad-42b3-8d29-a30b7bb2ae5a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:47:59,374::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:47:59,377::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:47:59,377::task::1191::Storage.TaskManager.Task::(prepare) Task=`c4d19fb1-9bad-42b3-8d29-a30b7bb2ae5a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:47:59,377::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4d19fb1-9bad-42b3-8d29-a30b7bb2ae5a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:47:59,377::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:47:59,377::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:47:59,377::task::993::Storage.TaskManager.Task::(_decref) Task=`c4d19fb1-9bad-42b3-8d29-a30b7bb2ae5a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:47:59,405::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:48:00,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:00,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913158 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:02,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:02,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912459 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:48:03,427::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:48:03,428::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:48:04,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:04,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913109 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:48:06,166::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:48:06,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655676 s, 798 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:48:06,228::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:48:06,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066853 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:06,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:06,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0928343 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:08,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:09,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548895 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:11,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:11,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914613 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:48:13,439::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:48:13,441::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:48:13,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:13,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913288 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:48:14,418::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:48:15,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:16,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0983649 s, 10.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:48:16,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:48:16,196::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000734402 s, 712 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:48:16,243::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:48:16,256::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069789 s, 532 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:48:17,855::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:48:18,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:19,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36946 s, 748 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:21,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:21,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913728 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:48:23,452::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:48:23,453::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:48:23,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:23,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091389 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:25,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:25,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914028 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:48:26,205::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:48:26,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000638852 s, 819 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:48:26,261::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:48:26,272::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675815 s, 549 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:27,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:28,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914107 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:48:29,430::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:48:30,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:30,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912256 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:32,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:32,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913791 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:48:33,464::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:48:33,465::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:48:34,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:34,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914517 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:48:36,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:48:36,234::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679229 s, 770 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:48:36,276::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:48:36,288::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650004 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:36,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:36,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915515 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:38,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:38,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091288 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:40,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:40,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915381 s, 11.2 MB/s\n'; <rc> = 0 >Detector thread::DEBUG::2014-07-14 14:48:42,266::protocoldetector::159::vds.MultiProtocolAcceptor::(_add_connection) Adding connection from 10.35.4.70:36984 >Detector thread::DEBUG::2014-07-14 14:48:42,268::protocoldetector::170::vds.MultiProtocolAcceptor::(_remove_connection) Connection removed from 10.35.4.70:36984 >Detector thread::DEBUG::2014-07-14 14:48:42,268::protocoldetector::196::vds.MultiProtocolAcceptor::(_handle_connection_read) Detected protocol xml from 10.35.4.70:36984 >Detector thread::DEBUG::2014-07-14 14:48:42,268::BindingXMLRPC::1147::XmlDetector::(handleSocket) xml over http detected from ('10.35.4.70', 36984) >Thread-11141::DEBUG::2014-07-14 14:48:42,271::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec020d5c-f1db-45a6-9304-c4cf7ccddffd`::moving from state init -> state preparing >Thread-11141::INFO::2014-07-14 14:48:42,271::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11141::INFO::2014-07-14 14:48:42,271::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650004', 'lastCheck': '6.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679229', 'lastCheck': '6.0', 'valid': True}} >Thread-11141::DEBUG::2014-07-14 14:48:42,271::task::1191::Storage.TaskManager.Task::(prepare) Task=`ec020d5c-f1db-45a6-9304-c4cf7ccddffd`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650004', 'lastCheck': '6.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679229', 'lastCheck': '6.0', 'valid': True}} >Thread-11141::DEBUG::2014-07-14 14:48:42,272::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec020d5c-f1db-45a6-9304-c4cf7ccddffd`::moving from state preparing -> state finished >Thread-11141::DEBUG::2014-07-14 14:48:42,272::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11141::DEBUG::2014-07-14 14:48:42,272::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11141::DEBUG::2014-07-14 14:48:42,272::task::993::Storage.TaskManager.Task::(_decref) Task=`ec020d5c-f1db-45a6-9304-c4cf7ccddffd`::ref 0 aborting False >Thread-11141::DEBUG::2014-07-14 14:48:42,400::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call getCapabilities with () {} >Thread-11141::DEBUG::2014-07-14 14:48:42,401::utils::717::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) >Thread-11141::DEBUG::2014-07-14 14:48:42,413::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-11141::DEBUG::2014-07-14 14:48:42,441::BindingXMLRPC::1114::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:d2dedd7a362'}], 'FC': []}, 'packages2': {'kernel': {'release': '301.fc20.x86_64', 'buildtime': 1386244877.0, 'version': '3.11.10'}, 'spice-server': {'release': '3.fc20', 'buildtime': 1383129969L, 'version': '0.12.4'}, 'vdsm': {'release': '136.git02f4fd5.fc19', 'buildtime': 1403104797L, 'version': '4.15.0'}, 'qemu-kvm': {'release': '4.fc20', 'buildtime': 1398798279L, 'version': '1.6.2'}, 'libvirt': {'release': '4.fc20', 'buildtime': 1395169169L, 'version': '1.1.3.4'}, 'qemu-img': {'release': '4.fc20', 'buildtime': 1398798279L, 'version': '1.6.2'}, 'mom': {'release': '1.master.fc19', 'buildtime': 1403103991L, 'version': '0.4.0'}}, 'numaNodeDistance': {'0': [10]}, 'cpuModel': 'Intel(R) Core(TM)2 Quad CPU Q8400 @ 2.66GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'selinux': {'mode': '1'}, 'kdumpStatus': 0, 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '10.35.4.198', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'gateway': '10.35.5.254', 'bootproto4': 'dhcp', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv4addrs': ['10.35.4.198/23'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.4.198', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv4addrs': ['10.35.4.198/23'], 'ipv6gateway': '::', 'gateway': '10.35.5.254', 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '174', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.180373ae5fe3', 'bridge_id': '8000.180373ae5fe3', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '5874', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'multicast_query_interval': '12500', 'multicast_query_use_ifaddr': '0', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}, 'ports': ['em1']}}, 'uuid': '44454C4C-3800-1056-8032-C8C04F34354A', 'lastClientIface': 'ovirtmgmt', 'nics': {'p2p2': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'p2p2', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '7ab5c17e-4d3d-4d9b-86ad-31e8e31eb4be', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': '00:07:E9:A5:1A:AF', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:07:e9:a5:1a:af', 'speed': 0}, 'em1': {'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '18:03:73:ae:5f:e3', 'DEVICE': 'em1', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '18:03:73:ae:5f:e3', 'speed': 100}, 'p2p1': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'p2p1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '77c39be3-75f9-438a-b26a-ca3253e979fb', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': '00:07:E9:A5:1A:AE', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:07:e9:a5:1a:ae', 'speed': 0}}, 'software_revision': '136', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,xsave,lahf_lm,dtherm,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:d2dedd7a362', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'autoNumaBalancing': 2, 'reservedMem': '321', 'bondings': {'bond0': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '5a:15:34:d5:9a:79', 'slaves': [], 'options': {}}}, 'software_version': '4.15', 'memSize': '7886', 'cpuSpeed': '2660.123', 'numaNodes': {u'0': {'totalMemory': '7886', 'cpus': [0, 1, 2, 3]}}, 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'pc-q35-1.4', u'pc-q35-1.5', u'q35', u'isapc', u'pc-0.10', u'pc-0.11', u'pc-0.12', u'pc-0.13', u'pc-0.14', u'pc-0.15', u'pc-1.0', u'pc-1.1', u'pc-1.2', u'pc-1.3', u'pc-i440fx-1.4', u'pc-i440fx-1.5', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '1', 'version': '20', 'name': 'Fedora'}, 'lastClient': '10.35.4.70'}} >Thread-11141::DEBUG::2014-07-14 14:48:42,470::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call getHardwareInfo with () {} >Thread-11141::DEBUG::2014-07-14 14:48:42,472::BindingXMLRPC::1114::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'OptiPlex 780', 'systemUUID': '44454c4c-3800-1056-8032-c8c04f34354a', 'systemSerialNumber': 'H8V245J', 'systemManufacturer': 'Dell Inc.'}} >Thread-11141::ERROR::2014-07-14 14:48:42,554::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:48:42,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:42,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913725 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:48:43,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:48:43,475::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:48:44,443::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:48:44,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:45,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913433 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:48:46,240::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:48:46,254::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064826 s, 807 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:48:46,292::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:48:46,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00509577 s, 72.8 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:47,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:47,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916717 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:48:47,891::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:48:49,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:49,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914786 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:51,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:51,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910086 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:53,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:48:53,487::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:48:53,488::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:48:53,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0954359 s, 10.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:55,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:55,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914362 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:48:56,266::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:48:56,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662813 s, 789 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:48:56,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:48:56,327::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066226 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:48:57,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:48:57,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914571 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11141::DEBUG::2014-07-14 14:48:58,380::task::595::Storage.TaskManager.Task::(_updateState) Task=`f299a1d9-56fe-463c-9050-34e0c1a4a956`::moving from state init -> state preparing >Thread-11141::INFO::2014-07-14 14:48:58,380::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11141::INFO::2014-07-14 14:48:58,381::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00066226', 'lastCheck': '2.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000662813', 'lastCheck': '2.1', 'valid': True}} >Thread-11141::DEBUG::2014-07-14 14:48:58,381::task::1191::Storage.TaskManager.Task::(prepare) Task=`f299a1d9-56fe-463c-9050-34e0c1a4a956`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00066226', 'lastCheck': '2.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000662813', 'lastCheck': '2.1', 'valid': True}} >Thread-11141::DEBUG::2014-07-14 14:48:58,381::task::595::Storage.TaskManager.Task::(_updateState) Task=`f299a1d9-56fe-463c-9050-34e0c1a4a956`::moving from state preparing -> state finished >Thread-11141::DEBUG::2014-07-14 14:48:58,381::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11141::DEBUG::2014-07-14 14:48:58,381::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11141::DEBUG::2014-07-14 14:48:58,381::task::993::Storage.TaskManager.Task::(_decref) Task=`f299a1d9-56fe-463c-9050-34e0c1a4a956`::ref 0 aborting False >Thread-11141::ERROR::2014-07-14 14:48:58,411::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:48:59,446::task::595::Storage.TaskManager.Task::(_updateState) Task=`75e10de2-b04f-4171-a303-f75fedfbf4bf`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:48:59,446::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:48:59,449::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:48:59,449::task::1191::Storage.TaskManager.Task::(prepare) Task=`75e10de2-b04f-4171-a303-f75fedfbf4bf`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:48:59,449::task::595::Storage.TaskManager.Task::(_updateState) Task=`75e10de2-b04f-4171-a303-f75fedfbf4bf`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:48:59,449::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:48:59,449::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:48:59,449::task::993::Storage.TaskManager.Task::(_decref) Task=`75e10de2-b04f-4171-a303-f75fedfbf4bf`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:48:59,476::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:48:59,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:00,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549969 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:02,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:02,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54235 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:49:03,498::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:49:03,500::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:49:04,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:05,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543657 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:49:06,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:49:06,294::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070518 s, 742 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:49:06,330::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:49:06,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637072 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:07,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:07,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912827 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:09,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:10,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559762 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:12,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:12,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918954 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:49:13,507::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:49:13,508::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Detector thread::DEBUG::2014-07-14 14:49:13,628::protocoldetector::159::vds.MultiProtocolAcceptor::(_add_connection) Adding connection from 10.35.4.70:33154 >Detector thread::DEBUG::2014-07-14 14:49:13,629::protocoldetector::170::vds.MultiProtocolAcceptor::(_remove_connection) Connection removed from 10.35.4.70:33154 >Detector thread::DEBUG::2014-07-14 14:49:13,630::protocoldetector::196::vds.MultiProtocolAcceptor::(_handle_connection_read) Detected protocol xml from 10.35.4.70:33154 >Detector thread::DEBUG::2014-07-14 14:49:13,630::BindingXMLRPC::1147::XmlDetector::(handleSocket) xml over http detected from ('10.35.4.70', 33154) >Thread-11142::DEBUG::2014-07-14 14:49:13,632::task::595::Storage.TaskManager.Task::(_updateState) Task=`b1393b9b-6287-41ac-8fd2-c89556f9e4d7`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:49:13,633::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:49:13,633::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000637072', 'lastCheck': '7.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00070518', 'lastCheck': '7.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:49:13,633::task::1191::Storage.TaskManager.Task::(prepare) Task=`b1393b9b-6287-41ac-8fd2-c89556f9e4d7`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000637072', 'lastCheck': '7.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00070518', 'lastCheck': '7.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:49:13,633::task::595::Storage.TaskManager.Task::(_updateState) Task=`b1393b9b-6287-41ac-8fd2-c89556f9e4d7`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:49:13,634::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:49:13,634::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:49:13,635::task::993::Storage.TaskManager.Task::(_decref) Task=`b1393b9b-6287-41ac-8fd2-c89556f9e4d7`::ref 0 aborting False >Thread-11142::DEBUG::2014-07-14 14:49:13,757::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call getCapabilities with () {} >Thread-11142::DEBUG::2014-07-14 14:49:13,759::utils::717::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) >Thread-11142::DEBUG::2014-07-14 14:49:13,769::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:49:13,795::BindingXMLRPC::1114::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:d2dedd7a362'}], 'FC': []}, 'packages2': {'kernel': {'release': '301.fc20.x86_64', 'buildtime': 1386244877.0, 'version': '3.11.10'}, 'spice-server': {'release': '3.fc20', 'buildtime': 1383129969L, 'version': '0.12.4'}, 'vdsm': {'release': '136.git02f4fd5.fc19', 'buildtime': 1403104797L, 'version': '4.15.0'}, 'qemu-kvm': {'release': '4.fc20', 'buildtime': 1398798279L, 'version': '1.6.2'}, 'libvirt': {'release': '4.fc20', 'buildtime': 1395169169L, 'version': '1.1.3.4'}, 'qemu-img': {'release': '4.fc20', 'buildtime': 1398798279L, 'version': '1.6.2'}, 'mom': {'release': '1.master.fc19', 'buildtime': 1403103991L, 'version': '0.4.0'}}, 'numaNodeDistance': {'0': [10]}, 'cpuModel': 'Intel(R) Core(TM)2 Quad CPU Q8400 @ 2.66GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'selinux': {'mode': '1'}, 'kdumpStatus': 0, 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '10.35.4.198', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'gateway': '10.35.5.254', 'bootproto4': 'dhcp', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv4addrs': ['10.35.4.198/23'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.4.198', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv4addrs': ['10.35.4.198/23'], 'ipv6gateway': '::', 'gateway': '10.35.5.254', 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '39', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.180373ae5fe3', 'bridge_id': '8000.180373ae5fe3', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '2739', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'multicast_query_interval': '12500', 'multicast_query_use_ifaddr': '0', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}, 'ports': ['em1']}}, 'uuid': '44454C4C-3800-1056-8032-C8C04F34354A', 'lastClientIface': 'ovirtmgmt', 'nics': {'p2p2': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'p2p2', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '7ab5c17e-4d3d-4d9b-86ad-31e8e31eb4be', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': '00:07:E9:A5:1A:AF', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:07:e9:a5:1a:af', 'speed': 0}, 'em1': {'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '18:03:73:ae:5f:e3', 'DEVICE': 'em1', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '18:03:73:ae:5f:e3', 'speed': 100}, 'p2p1': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'p2p1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '77c39be3-75f9-438a-b26a-ca3253e979fb', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': '00:07:E9:A5:1A:AE', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:07:e9:a5:1a:ae', 'speed': 0}}, 'software_revision': '136', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,xsave,lahf_lm,dtherm,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:d2dedd7a362', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'autoNumaBalancing': 2, 'reservedMem': '321', 'bondings': {'bond0': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '5a:15:34:d5:9a:79', 'slaves': [], 'options': {}}}, 'software_version': '4.15', 'memSize': '7886', 'cpuSpeed': '2660.123', 'numaNodes': {u'0': {'totalMemory': '7886', 'cpus': [0, 1, 2, 3]}}, 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'pc-q35-1.4', u'pc-q35-1.5', u'q35', u'isapc', u'pc-0.10', u'pc-0.11', u'pc-0.12', u'pc-0.13', u'pc-0.14', u'pc-0.15', u'pc-1.0', u'pc-1.1', u'pc-1.2', u'pc-1.3', u'pc-i440fx-1.4', u'pc-i440fx-1.5', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '1', 'version': '20', 'name': 'Fedora'}, 'lastClient': '10.35.4.70'}} >Thread-11142::DEBUG::2014-07-14 14:49:13,825::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call getHardwareInfo with () {} >Thread-11142::DEBUG::2014-07-14 14:49:13,826::BindingXMLRPC::1114::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'OptiPlex 780', 'systemUUID': '44454c4c-3800-1056-8032-c8c04f34354a', 'systemSerialNumber': 'H8V245J', 'systemManufacturer': 'Dell Inc.'}} >Thread-11142::ERROR::2014-07-14 14:49:13,968::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:49:14,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:14,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914152 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:49:14,488::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 14:49:16,300::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:49:16,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000736282 s, 710 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:49:16,346::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:49:16,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000609684 s, 609 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:16,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:16,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.096813 s, 10.6 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:49:17,921::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:49:18,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:18,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913386 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:20,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:20,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913903 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:22,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:22,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091434 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:49:23,521::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:49:23,522::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:49:24,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:25,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912706 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:49:26,321::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:49:26,333::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694339 s, 753 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:49:26,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:49:26,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639583 s, 580 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:27,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:27,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913566 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:29,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:49:29,501::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:49:29,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54854 s, 1.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:49:29,747::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d29ad87-743c-45b3-ad1f-36b08442a72f`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:49:29,747::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:49:29,747::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000639583', 'lastCheck': '3.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000694339', 'lastCheck': '3.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:49:29,748::task::1191::Storage.TaskManager.Task::(prepare) Task=`2d29ad87-743c-45b3-ad1f-36b08442a72f`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000639583', 'lastCheck': '3.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000694339', 'lastCheck': '3.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:49:29,748::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d29ad87-743c-45b3-ad1f-36b08442a72f`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:49:29,748::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:49:29,748::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:49:29,748::task::993::Storage.TaskManager.Task::(_decref) Task=`2d29ad87-743c-45b3-ad1f-36b08442a72f`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:49:29,780::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:49:31,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:31,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925263 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:49:33,533::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:49:33,534::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:49:33,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:33,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924278 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:35,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:36,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914841 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:49:36,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:49:36,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669589 s, 781 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:49:36,379::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:49:36,391::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676762 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:38,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:38,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540089 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:40,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:40,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923071 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:42,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:42,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914173 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:49:43,545::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:49:43,546::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:49:44,513::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:49:44,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:44,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914449 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:49:44,980::task::595::Storage.TaskManager.Task::(_updateState) Task=`d9d70d15-d194-4055-b26f-7fb9e79c649a`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:49:44,981::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:49:44,982::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000676762', 'lastCheck': '8.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000669589', 'lastCheck': '8.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:49:44,983::task::1191::Storage.TaskManager.Task::(prepare) Task=`d9d70d15-d194-4055-b26f-7fb9e79c649a`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000676762', 'lastCheck': '8.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000669589', 'lastCheck': '8.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:49:44,983::task::595::Storage.TaskManager.Task::(_updateState) Task=`d9d70d15-d194-4055-b26f-7fb9e79c649a`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:49:44,984::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:49:44,984::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:49:44,985::task::993::Storage.TaskManager.Task::(_decref) Task=`d9d70d15-d194-4055-b26f-7fb9e79c649a`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:49:45,019::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:49:46,355::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:49:46,368::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699226 s, 748 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:49:46,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:49:46,408::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000597261 s, 621 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:47,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:47,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.506766 s, 2.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:49:47,955::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:49:49,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:49,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913065 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:51,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:52,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.534513 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:49:53,550::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:49:53,551::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:49:54,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:54,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914233 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:56,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:49:56,442::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:49:56,442::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:49:56,468::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000614436 s, 851 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:49:56,472::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654379 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:56,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.134435 s, 7.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:49:58,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:49:58,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912604 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:49:59,516::task::595::Storage.TaskManager.Task::(_updateState) Task=`4db18317-690e-4ca5-8379-8faedc9b9015`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:49:59,517::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:49:59,520::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:49:59,520::task::1191::Storage.TaskManager.Task::(prepare) Task=`4db18317-690e-4ca5-8379-8faedc9b9015`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:49:59,520::task::595::Storage.TaskManager.Task::(_updateState) Task=`4db18317-690e-4ca5-8379-8faedc9b9015`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:49:59,520::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:49:59,520::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:49:59,520::task::993::Storage.TaskManager.Task::(_decref) Task=`4db18317-690e-4ca5-8379-8faedc9b9015`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:49:59,557::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-11142::DEBUG::2014-07-14 14:50:00,187::task::595::Storage.TaskManager.Task::(_updateState) Task=`d3d21f11-71f6-4e1c-8078-7d31aa51fadf`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:50:00,187::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:50:00,187::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000654379', 'lastCheck': '3.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000614436', 'lastCheck': '3.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:50:00,188::task::1191::Storage.TaskManager.Task::(prepare) Task=`d3d21f11-71f6-4e1c-8078-7d31aa51fadf`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000654379', 'lastCheck': '3.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000614436', 'lastCheck': '3.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:50:00,188::task::595::Storage.TaskManager.Task::(_updateState) Task=`d3d21f11-71f6-4e1c-8078-7d31aa51fadf`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:50:00,188::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:50:00,188::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:50:00,188::task::993::Storage.TaskManager.Task::(_decref) Task=`d3d21f11-71f6-4e1c-8078-7d31aa51fadf`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:50:00,202::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:00,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:00,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913922 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:02,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:02,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914633 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:50:03,562::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:50:03,563::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:04,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:05,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.516257 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:50:06,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:50:06,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:50:06,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00290739 s, 128 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:50:06,506::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630638 s, 829 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:07,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:07,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913355 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:09,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:10,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546827 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:12,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:12,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909995 s, 11.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:50:13,574::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:50:13,576::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:14,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:14,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915386 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:50:14,569::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-11142::DEBUG::2014-07-14 14:50:15,314::task::595::Storage.TaskManager.Task::(_updateState) Task=`24406123-dc5f-4466-b4b7-9d68a69c3c7f`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:50:15,315::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:50:15,315::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00290739', 'lastCheck': '8.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000630638', 'lastCheck': '8.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:50:15,315::task::1191::Storage.TaskManager.Task::(prepare) Task=`24406123-dc5f-4466-b4b7-9d68a69c3c7f`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00290739', 'lastCheck': '8.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000630638', 'lastCheck': '8.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:50:15,315::task::595::Storage.TaskManager.Task::(_updateState) Task=`24406123-dc5f-4466-b4b7-9d68a69c3c7f`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:50:15,315::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:50:15,315::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:50:15,316::task::993::Storage.TaskManager.Task::(_decref) Task=`24406123-dc5f-4466-b4b7-9d68a69c3c7f`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:50:15,329::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:16,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:16,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100953 s, 10.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:50:16,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:50:16,524::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:50:16,540::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000619159 s, 599 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:50:16,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675114 s, 775 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:50:17,986::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:50:18,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:18,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913538 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:20,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:20,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914288 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:22,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:22,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913837 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:50:23,587::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:50:23,588::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:24,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:25,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554918 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:50:26,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:50:26,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:50:26,574::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000717066 s, 517 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:50:26,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636852 s, 821 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:27,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:27,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913194 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:50:29,582::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:50:29,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:29,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911887 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:50:30,494::task::595::Storage.TaskManager.Task::(_updateState) Task=`63e80aa8-962e-45c3-b70d-b058c464529b`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:50:30,494::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:50:30,494::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000717066', 'lastCheck': '3.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000636852', 'lastCheck': '3.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:50:30,494::task::1191::Storage.TaskManager.Task::(prepare) Task=`63e80aa8-962e-45c3-b70d-b058c464529b`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000717066', 'lastCheck': '3.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000636852', 'lastCheck': '3.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:50:30,495::task::595::Storage.TaskManager.Task::(_updateState) Task=`63e80aa8-962e-45c3-b70d-b058c464529b`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:50:30,495::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:50:30,495::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:50:30,495::task::993::Storage.TaskManager.Task::(_decref) Task=`63e80aa8-962e-45c3-b70d-b058c464529b`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:50:30,516::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:31,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:31,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914112 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:50:33,599::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:50:33,600::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:33,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:33,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914323 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:35,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:36,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914404 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:50:36,580::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:50:36,592::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:50:36,608::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00127487 s, 291 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:50:36,609::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000723924 s, 722 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:38,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:38,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915327 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:40,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:40,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914161 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:42,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:42,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912423 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:50:43,608::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:50:43,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:44,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:44,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912959 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:50:44,594::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-11142::DEBUG::2014-07-14 14:50:45,657::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6f18911-5cd8-4f9c-9bf9-4bee38a183eb`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:50:45,658::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:50:45,660::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00127487', 'lastCheck': '9.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000723924', 'lastCheck': '9.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:50:45,660::task::1191::Storage.TaskManager.Task::(prepare) Task=`f6f18911-5cd8-4f9c-9bf9-4bee38a183eb`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00127487', 'lastCheck': '9.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000723924', 'lastCheck': '9.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:50:45,661::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6f18911-5cd8-4f9c-9bf9-4bee38a183eb`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:50:45,662::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:50:45,662::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:50:45,663::task::993::Storage.TaskManager.Task::(_decref) Task=`f6f18911-5cd8-4f9c-9bf9-4bee38a183eb`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:50:45,690::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:46,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:50:46,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:50:46,650::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:50:46,679::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663667 s, 788 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:50:46,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687183 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:46,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.137069 s, 7.5 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:50:48,018::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:50:48,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:48,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915677 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:50,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:51,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548307 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:53,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:53,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913363 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:50:53,620::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:50:53,621::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:50:55,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:50:56,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:50:56,899::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:50:56,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00387568 s, 95.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:50:56,916::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000727701 s, 719 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:56,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.40072 s, 731 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:50:58,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:50:59,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913657 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:50:59,598::task::595::Storage.TaskManager.Task::(_updateState) Task=`8895cab1-0fc5-4e1a-931c-a31ec930b14b`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:50:59,598::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:50:59,601::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:50:59,601::task::1191::Storage.TaskManager.Task::(prepare) Task=`8895cab1-0fc5-4e1a-931c-a31ec930b14b`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:50:59,601::task::595::Storage.TaskManager.Task::(_updateState) Task=`8895cab1-0fc5-4e1a-931c-a31ec930b14b`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:50:59,601::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:50:59,601::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:50:59,602::task::993::Storage.TaskManager.Task::(_decref) Task=`8895cab1-0fc5-4e1a-931c-a31ec930b14b`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:50:59,629::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-11142::DEBUG::2014-07-14 14:51:00,844::task::595::Storage.TaskManager.Task::(_updateState) Task=`9ecccebe-8aba-4d94-8837-6c8ef8cbdffd`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:51:00,844::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:51:00,845::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00387568', 'lastCheck': '3.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000727701', 'lastCheck': '3.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:51:00,845::task::1191::Storage.TaskManager.Task::(prepare) Task=`9ecccebe-8aba-4d94-8837-6c8ef8cbdffd`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00387568', 'lastCheck': '3.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000727701', 'lastCheck': '3.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:51:00,845::task::595::Storage.TaskManager.Task::(_updateState) Task=`9ecccebe-8aba-4d94-8837-6c8ef8cbdffd`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:51:00,845::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:51:00,845::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:51:00,846::task::993::Storage.TaskManager.Task::(_decref) Task=`9ecccebe-8aba-4d94-8837-6c8ef8cbdffd`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:51:00,865::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:51:01,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:01,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909036 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:03,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:51:03,633::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:51:03,634::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:51:03,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510489 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:05,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:06,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558527 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:51:06,924::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:51:06,937::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000713882 s, 520 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:51:06,940::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:51:06,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000784264 s, 667 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:08,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:08,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544209 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:10,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:10,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913577 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:12,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:13,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0931376 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:51:13,645::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:51:13,647::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:51:14,642::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:51:15,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:15,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913692 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:51:16,034::task::595::Storage.TaskManager.Task::(_updateState) Task=`10e9c428-df21-4349-b284-232f1fcf2d7f`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:51:16,035::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:51:16,036::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000713882', 'lastCheck': '9.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000784264', 'lastCheck': '9.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:51:16,037::task::1191::Storage.TaskManager.Task::(prepare) Task=`10e9c428-df21-4349-b284-232f1fcf2d7f`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000713882', 'lastCheck': '9.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000784264', 'lastCheck': '9.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:51:16,038::task::595::Storage.TaskManager.Task::(_updateState) Task=`10e9c428-df21-4349-b284-232f1fcf2d7f`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:51:16,038::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:51:16,039::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:51:16,040::task::993::Storage.TaskManager.Task::(_decref) Task=`10e9c428-df21-4349-b284-232f1fcf2d7f`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:51:16,072::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:51:16,955::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:51:16,967::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643462 s, 577 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:51:16,968::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:51:16,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000741142 s, 706 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:17,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:17,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557643 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:51:18,055::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:51:19,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:19,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912261 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:21,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:22,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913748 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:51:23,658::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:51:23,659::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:51:24,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:24,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913482 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:26,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:26,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913374 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:51:26,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:51:26,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:51:27,010::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00060434 s, 614 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:51:27,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0007367 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:28,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:28,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558259 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:51:29,655::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:51:30,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:30,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909659 s, 11.3 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:51:31,177::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcfc59db-87ba-4de2-a60d-1d2780eaa207`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:51:31,177::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:51:31,177::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00060434', 'lastCheck': '4.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0007367', 'lastCheck': '4.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:51:31,177::task::1191::Storage.TaskManager.Task::(prepare) Task=`fcfc59db-87ba-4de2-a60d-1d2780eaa207`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00060434', 'lastCheck': '4.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0007367', 'lastCheck': '4.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:51:31,178::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcfc59db-87ba-4de2-a60d-1d2780eaa207`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:51:31,178::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:51:31,178::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:51:31,178::task::993::Storage.TaskManager.Task::(_decref) Task=`fcfc59db-87ba-4de2-a60d-1d2780eaa207`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:51:31,196::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:51:32,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:33,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927092 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:51:33,670::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:51:33,671::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:51:35,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:35,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911922 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:51:37,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:51:37,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:51:37,040::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00518661 s, 71.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:51:37,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690398 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:37,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:37,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919654 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:39,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:39,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559978 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:41,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:42,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912066 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:51:43,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:51:43,683::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:51:44,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:44,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913678 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:51:44,668::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:51:46,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:46,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0983027 s, 10.4 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:51:46,290::task::595::Storage.TaskManager.Task::(_updateState) Task=`78774a99-9404-4a7e-90f6-68cf481d1c96`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:51:46,291::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:51:46,292::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00518661', 'lastCheck': '9.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000690398', 'lastCheck': '9.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:51:46,293::task::1191::Storage.TaskManager.Task::(prepare) Task=`78774a99-9404-4a7e-90f6-68cf481d1c96`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00518661', 'lastCheck': '9.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000690398', 'lastCheck': '9.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:51:46,294::task::595::Storage.TaskManager.Task::(_updateState) Task=`78774a99-9404-4a7e-90f6-68cf481d1c96`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:51:46,294::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:51:46,295::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:51:46,295::task::993::Storage.TaskManager.Task::(_decref) Task=`78774a99-9404-4a7e-90f6-68cf481d1c96`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:51:46,320::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:51:47,045::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:51:47,056::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:51:47,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000611663 s, 607 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:51:47,073::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640771 s, 816 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:51:48,088::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:51:48,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:48,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912803 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:50,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:50,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926587 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:52,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:52,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912741 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:51:53,694::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:51:53,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:51:54,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:55,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913354 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:51:57,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:51:57,088::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:51:57,100::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000859131 s, 432 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:51:57,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00117411 s, 445 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:51:57,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:51:58,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.33406 s, 768 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:51:59,671::task::595::Storage.TaskManager.Task::(_updateState) Task=`71316228-b785-43a1-9778-cb4c90e44220`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:51:59,672::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:51:59,674::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:51:59,674::task::1191::Storage.TaskManager.Task::(prepare) Task=`71316228-b785-43a1-9778-cb4c90e44220`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:51:59,675::task::595::Storage.TaskManager.Task::(_updateState) Task=`71316228-b785-43a1-9778-cb4c90e44220`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:51:59,675::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:51:59,675::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:51:59,675::task::993::Storage.TaskManager.Task::(_decref) Task=`71316228-b785-43a1-9778-cb4c90e44220`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:51:59,702::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:52:00,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:01,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.517168 s, 2.0 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:52:01,423::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d787156-df08-4b42-9e42-fe02f1718bd9`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:52:01,423::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:52:01,424::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000859131', 'lastCheck': '4.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00117411', 'lastCheck': '4.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:52:01,424::task::1191::Storage.TaskManager.Task::(prepare) Task=`0d787156-df08-4b42-9e42-fe02f1718bd9`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000859131', 'lastCheck': '4.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00117411', 'lastCheck': '4.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:52:01,424::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d787156-df08-4b42-9e42-fe02f1718bd9`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:52:01,424::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:52:01,424::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:52:01,425::task::993::Storage.TaskManager.Task::(_decref) Task=`0d787156-df08-4b42-9e42-fe02f1718bd9`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:52:01,437::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:52:03,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:52:03,706::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:52:03,707::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:52:03,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544423 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:05,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:06,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913851 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:52:07,106::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:52:07,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:52:07,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621895 s, 597 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:52:07,130::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671971 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:08,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:08,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55871 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:10,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:11,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507362 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:13,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:13,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911782 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:52:13,718::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:52:13,719::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:52:14,714::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:52:15,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:15,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913654 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:52:16,532::task::595::Storage.TaskManager.Task::(_updateState) Task=`327aaf71-6161-42df-b9d8-6445f95c874c`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:52:16,532::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:52:16,533::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000621895', 'lastCheck': '9.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000671971', 'lastCheck': '9.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:52:16,534::task::1191::Storage.TaskManager.Task::(prepare) Task=`327aaf71-6161-42df-b9d8-6445f95c874c`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000621895', 'lastCheck': '9.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000671971', 'lastCheck': '9.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:52:16,535::task::595::Storage.TaskManager.Task::(_updateState) Task=`327aaf71-6161-42df-b9d8-6445f95c874c`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:52:16,536::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:52:16,536::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:52:16,537::task::993::Storage.TaskManager.Task::(_decref) Task=`327aaf71-6161-42df-b9d8-6445f95c874c`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:52:16,561::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:52:17,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:52:17,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:52:17,244::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685114 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:52:17,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675622 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:17,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:17,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912814 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:52:18,128::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:52:19,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:20,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540307 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:22,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:22,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544515 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:52:23,730::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:52:23,731::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:52:24,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:24,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912314 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:26,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:52:27,249::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:52:27,254::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:52:27,269::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705294 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:52:27,285::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:52:27,287::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Dummy-49::DEBUG::2014-07-14 14:52:27,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515016 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:52:27,291::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:52:27,297::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:52:27,298::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:52:27,298::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:52:27,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:52:27,316::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006112 s, 856 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:29,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:29,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913902 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:52:29,727::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:52:31,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:31,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913206 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:52:31,649::task::595::Storage.TaskManager.Task::(_updateState) Task=`44644480-2dd2-4514-9dee-781f489db2e9`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:52:31,649::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:52:31,650::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000705294', 'lastCheck': '4.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0006112', 'lastCheck': '4.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:52:31,650::task::1191::Storage.TaskManager.Task::(prepare) Task=`44644480-2dd2-4514-9dee-781f489db2e9`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000705294', 'lastCheck': '4.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0006112', 'lastCheck': '4.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:52:31,650::task::595::Storage.TaskManager.Task::(_updateState) Task=`44644480-2dd2-4514-9dee-781f489db2e9`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:52:31,650::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:52:31,650::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:52:31,650::task::993::Storage.TaskManager.Task::(_decref) Task=`44644480-2dd2-4514-9dee-781f489db2e9`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:52:31,663::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:52:33,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:33,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912495 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:52:33,741::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:52:33,742::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:52:35,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:36,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.536038 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:52:37,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:52:37,292::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637228 s, 582 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:52:37,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:52:37,335::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635117 s, 823 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:38,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:38,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913523 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:40,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:40,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913044 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:42,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:43,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557336 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:52:43,751::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:52:43,752::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:52:44,740::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:52:45,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:45,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913993 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:52:46,755::task::595::Storage.TaskManager.Task::(_updateState) Task=`ed68836e-e8c3-462c-b322-5fe028c1a041`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:52:46,755::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:52:46,755::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000637228', 'lastCheck': '9.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000635117', 'lastCheck': '9.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:52:46,756::task::1191::Storage.TaskManager.Task::(prepare) Task=`ed68836e-e8c3-462c-b322-5fe028c1a041`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000637228', 'lastCheck': '9.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000635117', 'lastCheck': '9.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:52:46,756::task::595::Storage.TaskManager.Task::(_updateState) Task=`ed68836e-e8c3-462c-b322-5fe028c1a041`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:52:46,756::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:52:46,756::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:52:46,756::task::993::Storage.TaskManager.Task::(_decref) Task=`ed68836e-e8c3-462c-b322-5fe028c1a041`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:52:46,770::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:52:47,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:47,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912554 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:52:47,299::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:52:47,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000609017 s, 609 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:52:47,338::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:52:47,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000644599 s, 811 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:52:48,159::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:52:49,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:49,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097946 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:51,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:51,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912357 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:53,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:53,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912725 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:52:53,761::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:52:53,762::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:52:55,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:55,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914024 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:52:57,317::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:52:57,330::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 14:52:57,331::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:52:57,334::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:52:57,336::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:52:57,336::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:52:57,336::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:52:57,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:52:57,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667195 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:52:57,353::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:52:57,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667625 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:52:57,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:52:57,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912829 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:52:59,744::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4d48b18-49ce-45bf-b44d-bd516a869965`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:52:59,744::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:52:59,747::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:52:59,747::task::1191::Storage.TaskManager.Task::(prepare) Task=`e4d48b18-49ce-45bf-b44d-bd516a869965`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:52:59,747::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4d48b18-49ce-45bf-b44d-bd516a869965`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:52:59,748::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:52:59,748::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:52:59,748::task::993::Storage.TaskManager.Task::(_decref) Task=`e4d48b18-49ce-45bf-b44d-bd516a869965`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:52:59,774::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:52:59,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:00,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913524 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:53:01,894::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c5e4e5c-7d80-44f0-b931-858ada0ac36b`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:53:01,895::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:53:01,895::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667195', 'lastCheck': '4.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667625', 'lastCheck': '4.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:53:01,895::task::1191::Storage.TaskManager.Task::(prepare) Task=`2c5e4e5c-7d80-44f0-b931-858ada0ac36b`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667195', 'lastCheck': '4.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667625', 'lastCheck': '4.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:53:01,895::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c5e4e5c-7d80-44f0-b931-858ada0ac36b`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:53:01,896::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:53:01,896::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:53:01,896::task::993::Storage.TaskManager.Task::(_decref) Task=`2c5e4e5c-7d80-44f0-b931-858ada0ac36b`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:53:01,914::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:53:02,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:02,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091184 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:53:03,773::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:53:03,774::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:53:04,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:04,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515382 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:06,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:06,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091441 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:53:07,353::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:53:07,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664458 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:53:07,369::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:53:07,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653047 s, 801 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:08,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:08,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914353 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:10,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:11,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911806 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:13,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:13,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914193 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:53:13,781::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:53:13,782::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:53:14,786::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:53:15,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:15,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915226 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:53:17,023::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a3bfa5d-ffe1-4d3b-b749-5e9f0bfc4066`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:53:17,024::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:53:17,024::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000664458', 'lastCheck': '9.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000653047', 'lastCheck': '9.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:53:17,024::task::1191::Storage.TaskManager.Task::(prepare) Task=`1a3bfa5d-ffe1-4d3b-b749-5e9f0bfc4066`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000664458', 'lastCheck': '9.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000653047', 'lastCheck': '9.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:53:17,024::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a3bfa5d-ffe1-4d3b-b749-5e9f0bfc4066`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:53:17,025::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:53:17,025::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:53:17,025::task::993::Storage.TaskManager.Task::(_decref) Task=`1a3bfa5d-ffe1-4d3b-b749-5e9f0bfc4066`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:53:17,037::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:53:17,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:53:17,428::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:53:17,442::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:53:17,443::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705213 s, 526 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:53:17,460::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000751933 s, 696 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:17,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129441 s, 7.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:53:18,192::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:53:19,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:19,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.1028 s, 10.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:21,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:21,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918259 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:53:23,791::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:53:23,792::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:53:23,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:23,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913341 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:25,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:26,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912096 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:53:27,463::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:53:27,475::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643153 s, 577 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:53:27,477::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:53:27,489::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665727 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:28,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:28,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913327 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:53:29,800::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:53:30,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:30,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091294 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:53:32,136::task::595::Storage.TaskManager.Task::(_updateState) Task=`4172debf-a9ef-4d2f-bfaa-dc99697948b6`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:53:32,136::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:53:32,137::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000643153', 'lastCheck': '4.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000665727', 'lastCheck': '4.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:53:32,137::task::1191::Storage.TaskManager.Task::(prepare) Task=`4172debf-a9ef-4d2f-bfaa-dc99697948b6`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000643153', 'lastCheck': '4.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000665727', 'lastCheck': '4.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:53:32,137::task::595::Storage.TaskManager.Task::(_updateState) Task=`4172debf-a9ef-4d2f-bfaa-dc99697948b6`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:53:32,137::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:53:32,137::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:53:32,137::task::993::Storage.TaskManager.Task::(_decref) Task=`4172debf-a9ef-4d2f-bfaa-dc99697948b6`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:53:32,150::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:53:32,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:32,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913492 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:53:33,803::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:53:33,804::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:53:34,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:34,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556569 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:36,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:53:37,510::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:53:37,521::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:53:37,535::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00534528 s, 97.8 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:53:37,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000812478 s, 457 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:37,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559521 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:39,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:39,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091427 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:41,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:41,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912816 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:43,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:53:43,816::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:53:43,817::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:53:43,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913665 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:53:44,818::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:53:45,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:46,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912606 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:53:47,288::task::595::Storage.TaskManager.Task::(_updateState) Task=`29c67589-960a-476f-b6e0-f28ab720843c`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:53:47,288::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:53:47,289::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000812478', 'lastCheck': '9.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00534528', 'lastCheck': '9.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:53:47,289::task::1191::Storage.TaskManager.Task::(prepare) Task=`29c67589-960a-476f-b6e0-f28ab720843c`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000812478', 'lastCheck': '9.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00534528', 'lastCheck': '9.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:53:47,289::task::595::Storage.TaskManager.Task::(_updateState) Task=`29c67589-960a-476f-b6e0-f28ab720843c`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:53:47,289::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:53:47,289::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:53:47,290::task::993::Storage.TaskManager.Task::(_decref) Task=`29c67589-960a-476f-b6e0-f28ab720843c`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:53:47,307::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:53:47,548::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:53:47,559::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:53:47,570::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661439 s, 561 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:53:47,572::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062478 s, 837 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:48,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 14:53:48,232::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:53:48,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548975 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:50,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:50,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091335 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:52,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:53,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912255 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:53:53,828::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:53:53,829::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:53:55,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:55,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546691 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:53:57,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:53:57,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:53:57,588::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000696939 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:53:57,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000761337 s, 687 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:53:57,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:53:58,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36493 s, 750 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:53:59,821::task::595::Storage.TaskManager.Task::(_updateState) Task=`d4298360-ae32-491d-af2d-03d5282ba9e5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:53:59,822::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:53:59,827::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:53:59,828::task::1191::Storage.TaskManager.Task::(prepare) Task=`d4298360-ae32-491d-af2d-03d5282ba9e5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:53:59,829::task::595::Storage.TaskManager.Task::(_updateState) Task=`d4298360-ae32-491d-af2d-03d5282ba9e5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:53:59,830::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:53:59,831::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:53:59,831::task::993::Storage.TaskManager.Task::(_decref) Task=`d4298360-ae32-491d-af2d-03d5282ba9e5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:53:59,861::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:54:00,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:01,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922301 s, 11.1 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:54:02,399::task::595::Storage.TaskManager.Task::(_updateState) Task=`5252cfa0-9c49-487c-93b1-0baed627b235`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:54:02,400::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:54:02,400::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000696939', 'lastCheck': '4.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000761337', 'lastCheck': '4.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:54:02,400::task::1191::Storage.TaskManager.Task::(prepare) Task=`5252cfa0-9c49-487c-93b1-0baed627b235`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000696939', 'lastCheck': '4.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000761337', 'lastCheck': '4.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:54:02,400::task::595::Storage.TaskManager.Task::(_updateState) Task=`5252cfa0-9c49-487c-93b1-0baed627b235`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:54:02,401::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:54:02,401::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:54:02,401::task::993::Storage.TaskManager.Task::(_decref) Task=`5252cfa0-9c49-487c-93b1-0baed627b235`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:54:02,413::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:54:03,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:03,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912478 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:54:03,837::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:54:03,838::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:54:05,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:05,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913403 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:07,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:07,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918904 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:54:07,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:54:07,616::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:54:07,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654674 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:54:07,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679532 s, 770 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:09,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:09,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914296 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:11,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:11,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913837 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:13,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:13,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914716 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:54:13,850::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:54:13,851::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:54:14,874::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:54:15,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:15,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913713 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:54:17,532::task::595::Storage.TaskManager.Task::(_updateState) Task=`07c167dd-50de-4731-9337-bf79a6ad1464`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:54:17,532::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:54:17,532::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000654674', 'lastCheck': '9.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679532', 'lastCheck': '9.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:54:17,532::task::1191::Storage.TaskManager.Task::(prepare) Task=`07c167dd-50de-4731-9337-bf79a6ad1464`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000654674', 'lastCheck': '9.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679532', 'lastCheck': '9.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:54:17,533::task::595::Storage.TaskManager.Task::(_updateState) Task=`07c167dd-50de-4731-9337-bf79a6ad1464`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:54:17,533::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:54:17,533::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:54:17,533::task::993::Storage.TaskManager.Task::(_decref) Task=`07c167dd-50de-4731-9337-bf79a6ad1464`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:54:17,545::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:54:17,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:54:17,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:54:17,657::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669216 s, 554 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:54:17,660::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703295 s, 744 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:18,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 14:54:18,265::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:54:18,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.530442 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:20,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:21,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36369 s, 751 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:54:23,861::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:54:23,863::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:54:23,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:24,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916461 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:26,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:26,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913751 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:28,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:54:28,432::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:54:28,432::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:54:28,523::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000711803 s, 735 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:54:28,523::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000872109 s, 425 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:28,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102179 s, 10.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:54:29,886::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:54:30,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:30,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914769 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:54:32,646::task::595::Storage.TaskManager.Task::(_updateState) Task=`079eea34-96ca-4331-a37f-c220f5d6095e`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:54:32,647::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:54:32,647::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000872109', 'lastCheck': '4.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000711803', 'lastCheck': '4.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:54:32,647::task::1191::Storage.TaskManager.Task::(prepare) Task=`079eea34-96ca-4331-a37f-c220f5d6095e`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000872109', 'lastCheck': '4.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000711803', 'lastCheck': '4.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:54:32,647::task::595::Storage.TaskManager.Task::(_updateState) Task=`079eea34-96ca-4331-a37f-c220f5d6095e`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:54:32,647::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:54:32,648::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:54:32,648::task::993::Storage.TaskManager.Task::(_decref) Task=`079eea34-96ca-4331-a37f-c220f5d6095e`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:54:32,660::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:54:32,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:33,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556973 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:54:33,874::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:54:33,875::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:54:35,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:35,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914125 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:37,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:37,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091537 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:54:38,532::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:54:38,543::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:54:38,555::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000694446 s, 534 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:54:38,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716155 s, 730 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:39,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:39,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913527 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:41,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:42,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36222 s, 752 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:54:43,881::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:54:43,883::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:54:44,897::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:54:44,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:45,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547673 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:47,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:47,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091371 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:54:47,764::task::595::Storage.TaskManager.Task::(_updateState) Task=`adf8a0e5-c9b3-4206-91ef-de4d7f5bf8cb`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:54:47,764::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:54:47,764::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000694446', 'lastCheck': '9.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000716155', 'lastCheck': '9.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:54:47,765::task::1191::Storage.TaskManager.Task::(prepare) Task=`adf8a0e5-c9b3-4206-91ef-de4d7f5bf8cb`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000694446', 'lastCheck': '9.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000716155', 'lastCheck': '9.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:54:47,765::task::595::Storage.TaskManager.Task::(_updateState) Task=`adf8a0e5-c9b3-4206-91ef-de4d7f5bf8cb`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:54:47,765::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:54:47,765::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:54:47,765::task::993::Storage.TaskManager.Task::(_decref) Task=`adf8a0e5-c9b3-4206-91ef-de4d7f5bf8cb`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:54:47,778::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >VM Channels Listener::DEBUG::2014-07-14 14:54:48,297::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 14:54:48,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:54:48,570::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:54:48,581::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657408 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:54:48,583::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676741 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:49,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:49,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914042 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:51,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:51,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912835 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:54:53,889::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:54:53,891::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:54:54,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:54,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.514775 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:56,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:56,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914242 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:54:58,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:54:58,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000694173 s, 534 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:54:58,601::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:54:58,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000708672 s, 738 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:54:58,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:54:58,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913699 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:54:59,901::task::595::Storage.TaskManager.Task::(_updateState) Task=`ecf7255f-9e8a-47ad-bb0b-f8c0922676d7`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:54:59,901::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:54:59,904::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:54:59,904::task::1191::Storage.TaskManager.Task::(prepare) Task=`ecf7255f-9e8a-47ad-bb0b-f8c0922676d7`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:54:59,904::task::595::Storage.TaskManager.Task::(_updateState) Task=`ecf7255f-9e8a-47ad-bb0b-f8c0922676d7`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:54:59,904::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:54:59,905::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:54:59,905::task::993::Storage.TaskManager.Task::(_decref) Task=`ecf7255f-9e8a-47ad-bb0b-f8c0922676d7`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:54:59,931::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:55:00,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:00,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938172 s, 10.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:55:02,873::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc4b75e5-8b26-495b-bc4b-8abd89d7ebc8`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:55:02,873::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:55:02,873::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000694173', 'lastCheck': '4.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000708672', 'lastCheck': '4.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:55:02,873::task::1191::Storage.TaskManager.Task::(prepare) Task=`dc4b75e5-8b26-495b-bc4b-8abd89d7ebc8`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000694173', 'lastCheck': '4.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000708672', 'lastCheck': '4.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:55:02,874::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc4b75e5-8b26-495b-bc4b-8abd89d7ebc8`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:55:02,874::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:55:02,874::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:55:02,874::task::993::Storage.TaskManager.Task::(_decref) Task=`dc4b75e5-8b26-495b-bc4b-8abd89d7ebc8`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:55:02,886::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:55:02,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:02,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912047 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:55:03,897::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:55:03,898::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:55:05,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:05,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545358 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:07,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:07,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913075 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:55:08,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:55:08,627::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:55:08,639::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000801948 s, 463 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:55:08,640::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000611212 s, 856 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:09,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:09,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917416 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:11,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:11,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913272 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:55:13,908::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:55:13,909::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:55:13,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:14,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914395 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:55:14,945::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:55:16,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:16,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54226 s, 1.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:55:17,980::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf05ee50-84b0-4408-812a-1fa05d526764`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:55:17,981::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:55:17,981::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000801948', 'lastCheck': '9.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000611212', 'lastCheck': '9.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:55:17,981::task::1191::Storage.TaskManager.Task::(prepare) Task=`bf05ee50-84b0-4408-812a-1fa05d526764`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000801948', 'lastCheck': '9.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000611212', 'lastCheck': '9.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:55:17,981::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf05ee50-84b0-4408-812a-1fa05d526764`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:55:17,981::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:55:17,982::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:55:17,982::task::993::Storage.TaskManager.Task::(_decref) Task=`bf05ee50-84b0-4408-812a-1fa05d526764`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:55:17,994::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >VM Channels Listener::DEBUG::2014-07-14 14:55:18,329::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:55:18,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:55:18,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:55:18,667::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:55:18,668::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657559 s, 795 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:55:18,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704524 s, 527 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:18,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10047 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:20,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:20,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912605 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:22,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:23,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913386 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:55:23,920::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:55:23,921::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:55:25,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:25,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922424 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:27,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:27,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914458 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:55:28,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:55:28,769::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:55:28,790::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000722985 s, 723 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:55:28,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642283 s, 578 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:29,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:29,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912585 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:55:29,957::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:55:31,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:31,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091521 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:55:33,103::task::595::Storage.TaskManager.Task::(_updateState) Task=`6a585781-180b-4974-b91b-8b8c64b6debc`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:55:33,104::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:55:33,104::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000642283', 'lastCheck': '4.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000722985', 'lastCheck': '4.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:55:33,104::task::1191::Storage.TaskManager.Task::(prepare) Task=`6a585781-180b-4974-b91b-8b8c64b6debc`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000642283', 'lastCheck': '4.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000722985', 'lastCheck': '4.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:55:33,104::task::595::Storage.TaskManager.Task::(_updateState) Task=`6a585781-180b-4974-b91b-8b8c64b6debc`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:55:33,105::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:55:33,105::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:55:33,105::task::993::Storage.TaskManager.Task::(_decref) Task=`6a585781-180b-4974-b91b-8b8c64b6debc`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:55:33,118::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:55:33,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:33,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913301 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:55:33,932::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:55:33,933::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:55:35,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:35,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914984 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:37,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:37,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913677 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:55:38,798::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:55:38,808::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:55:38,820::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00270176 s, 137 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:55:38,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000756925 s, 691 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:39,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:39,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912592 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:41,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:42,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925689 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:55:43,940::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:55:43,941::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:55:44,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:44,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914576 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:55:44,978::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:55:46,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:46,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968369 s, 10.6 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:55:48,219::task::595::Storage.TaskManager.Task::(_updateState) Task=`b593d102-877c-404f-848d-2f7c2aefd4ff`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:55:48,219::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:55:48,220::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00270176', 'lastCheck': '9.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000756925', 'lastCheck': '9.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:55:48,220::task::1191::Storage.TaskManager.Task::(prepare) Task=`b593d102-877c-404f-848d-2f7c2aefd4ff`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00270176', 'lastCheck': '9.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000756925', 'lastCheck': '9.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:55:48,220::task::595::Storage.TaskManager.Task::(_updateState) Task=`b593d102-877c-404f-848d-2f7c2aefd4ff`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:55:48,220::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:55:48,220::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:55:48,221::task::993::Storage.TaskManager.Task::(_decref) Task=`b593d102-877c-404f-848d-2f7c2aefd4ff`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:55:48,232::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >VM Channels Listener::DEBUG::2014-07-14 14:55:48,361::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:55:48,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:48,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091307 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:55:48,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:55:48,837::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:55:48,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000613673 s, 605 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:55:48,850::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00063407 s, 825 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:50,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:50,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912771 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:52,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:53,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.514079 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:55:53,952::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:55:53,954::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:55:55,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:55,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551408 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:55:57,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:55:58,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.520687 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:55:58,856::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:55:58,867::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:55:58,882::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686877 s, 540 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:55:58,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698287 s, 749 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:55:59,982::task::595::Storage.TaskManager.Task::(_updateState) Task=`c64b8fef-ac7f-4cce-a86d-dd429b88fcc8`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:55:59,982::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:55:59,986::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:55:59,986::task::1191::Storage.TaskManager.Task::(prepare) Task=`c64b8fef-ac7f-4cce-a86d-dd429b88fcc8`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:55:59,986::task::595::Storage.TaskManager.Task::(_updateState) Task=`c64b8fef-ac7f-4cce-a86d-dd429b88fcc8`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:55:59,986::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:55:59,986::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:55:59,986::task::993::Storage.TaskManager.Task::(_decref) Task=`c64b8fef-ac7f-4cce-a86d-dd429b88fcc8`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:56:00,013::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:56:00,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:00,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913119 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:02,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:02,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912892 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:56:03,338::task::595::Storage.TaskManager.Task::(_updateState) Task=`059a4306-5aa9-4406-a369-90a996fbca0b`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:56:03,339::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:56:03,339::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000686877', 'lastCheck': '4.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000698287', 'lastCheck': '4.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:56:03,339::task::1191::Storage.TaskManager.Task::(prepare) Task=`059a4306-5aa9-4406-a369-90a996fbca0b`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000686877', 'lastCheck': '4.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000698287', 'lastCheck': '4.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:56:03,339::task::595::Storage.TaskManager.Task::(_updateState) Task=`059a4306-5aa9-4406-a369-90a996fbca0b`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:56:03,340::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:56:03,340::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:56:03,340::task::993::Storage.TaskManager.Task::(_decref) Task=`059a4306-5aa9-4406-a369-90a996fbca0b`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:56:03,352::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:56:03,964::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:56:03,965::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:56:04,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:04,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913065 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:06,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:06,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091763 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:08,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 14:56:08,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:56:08,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:56:08,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00408437 s, 90.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:56:08,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000836596 s, 625 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:08,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.128204 s, 8.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:10,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:11,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550829 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:13,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:13,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912961 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:56:13,974::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:56:13,975::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:56:15,025::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:56:15,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:16,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550705 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:18,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 14:56:18,394::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-11142::DEBUG::2014-07-14 14:56:18,453::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf3ac25f-a170-4f97-9d86-22b1c0e1cbfd`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:56:18,453::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:56:18,453::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00408437', 'lastCheck': '9.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000836596', 'lastCheck': '9.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:56:18,454::task::1191::Storage.TaskManager.Task::(prepare) Task=`bf3ac25f-a170-4f97-9d86-22b1c0e1cbfd`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00408437', 'lastCheck': '9.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000836596', 'lastCheck': '9.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:56:18,454::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf3ac25f-a170-4f97-9d86-22b1c0e1cbfd`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:56:18,454::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:56:18,454::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:56:18,454::task::993::Storage.TaskManager.Task::(_decref) Task=`bf3ac25f-a170-4f97-9d86-22b1c0e1cbfd`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:56:18,469::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:56:18,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.501813 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:56:18,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:56:18,959::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:56:18,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00615991 s, 60.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:56:18,971::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663661 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:20,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:21,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548589 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:23,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:23,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913203 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:56:23,986::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:56:23,987::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:56:25,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:25,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918142 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:27,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:27,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913569 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:56:28,975::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:56:28,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:56:28,989::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684904 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:56:29,004::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696753 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:29,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:29,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912925 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:56:30,037::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:56:31,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:32,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912181 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:56:33,567::task::595::Storage.TaskManager.Task::(_updateState) Task=`4fe2bf59-a54f-45eb-8abb-ce06daf01a69`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:56:33,567::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:56:33,567::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000684904', 'lastCheck': '4.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000696753', 'lastCheck': '4.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:56:33,567::task::1191::Storage.TaskManager.Task::(prepare) Task=`4fe2bf59-a54f-45eb-8abb-ce06daf01a69`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000684904', 'lastCheck': '4.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000696753', 'lastCheck': '4.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:56:33,568::task::595::Storage.TaskManager.Task::(_updateState) Task=`4fe2bf59-a54f-45eb-8abb-ce06daf01a69`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:56:33,568::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:56:33,568::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:56:33,568::task::993::Storage.TaskManager.Task::(_decref) Task=`4fe2bf59-a54f-45eb-8abb-ce06daf01a69`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:56:33,581::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:56:34,005::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:56:34,006::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:56:34,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:34,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913173 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:36,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:36,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915108 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:38,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:38,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511958 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:56:39,007::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:56:39,020::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:56:39,036::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690931 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:56:39,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000628699 s, 832 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:40,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:40,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913291 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:42,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:43,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913017 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:56:44,023::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:56:44,024::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:56:45,050::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:56:45,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:45,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914336 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:47,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:47,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914232 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:56:48,427::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-11142::DEBUG::2014-07-14 14:56:48,691::task::595::Storage.TaskManager.Task::(_updateState) Task=`7cfea66e-6570-4a76-aa3b-f06ac450be66`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:56:48,691::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:56:48,692::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000690931', 'lastCheck': '9.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000628699', 'lastCheck': '9.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:56:48,692::task::1191::Storage.TaskManager.Task::(prepare) Task=`7cfea66e-6570-4a76-aa3b-f06ac450be66`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000690931', 'lastCheck': '9.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000628699', 'lastCheck': '9.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:56:48,692::task::595::Storage.TaskManager.Task::(_updateState) Task=`7cfea66e-6570-4a76-aa3b-f06ac450be66`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:56:48,692::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:56:48,692::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:56:48,692::task::993::Storage.TaskManager.Task::(_decref) Task=`7cfea66e-6570-4a76-aa3b-f06ac450be66`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:56:48,705::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:56:49,040::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:56:49,052::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:56:49,067::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00223728 s, 166 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:56:49,070::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677357 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:49,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:49,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518302 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:51,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:53,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37025 s, 747 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:56:54,035::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:56:54,036::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:56:55,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:55,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511063 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:57,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:56:57,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913604 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:56:59,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:56:59,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663637 s, 559 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:56:59,085::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:56:59,097::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677096 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:56:59,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:00,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915083 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:57:00,054::task::595::Storage.TaskManager.Task::(_updateState) Task=`af1acb82-8cc2-496e-8aa4-aef760e2c379`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:57:00,054::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:57:00,057::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:57:00,057::task::1191::Storage.TaskManager.Task::(prepare) Task=`af1acb82-8cc2-496e-8aa4-aef760e2c379`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:57:00,057::task::595::Storage.TaskManager.Task::(_updateState) Task=`af1acb82-8cc2-496e-8aa4-aef760e2c379`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:57:00,057::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:57:00,058::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:57:00,058::task::993::Storage.TaskManager.Task::(_decref) Task=`af1acb82-8cc2-496e-8aa4-aef760e2c379`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:57:00,085::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:57:02,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:02,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913146 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:57:03,789::task::595::Storage.TaskManager.Task::(_updateState) Task=`45e57115-2092-4582-869d-b79fd4cdac5a`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:57:03,790::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:57:03,790::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000663637', 'lastCheck': '4.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000677096', 'lastCheck': '4.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:57:03,790::task::1191::Storage.TaskManager.Task::(prepare) Task=`45e57115-2092-4582-869d-b79fd4cdac5a`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000663637', 'lastCheck': '4.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000677096', 'lastCheck': '4.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:57:03,790::task::595::Storage.TaskManager.Task::(_updateState) Task=`45e57115-2092-4582-869d-b79fd4cdac5a`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:57:03,791::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:57:03,791::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:57:03,791::task::993::Storage.TaskManager.Task::(_decref) Task=`45e57115-2092-4582-869d-b79fd4cdac5a`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:57:03,803::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:57:04,047::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:57:04,048::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:57:04,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:04,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.51375 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:06,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:07,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540715 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:57:09,098::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:57:09,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:57:09,111::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00076125 s, 487 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:57:09,124::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068848 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:09,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:09,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915232 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:11,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:11,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556834 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:13,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:14,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914566 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:57:14,057::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:57:14,058::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:57:15,098::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:57:16,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:16,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0972687 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:18,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:18,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913862 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:57:18,459::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-11142::DEBUG::2014-07-14 14:57:18,897::task::595::Storage.TaskManager.Task::(_updateState) Task=`304e7b85-ec83-414f-916b-7254e47dddac`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:57:18,898::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:57:18,898::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00076125', 'lastCheck': '9.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00068848', 'lastCheck': '9.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:57:18,898::task::1191::Storage.TaskManager.Task::(prepare) Task=`304e7b85-ec83-414f-916b-7254e47dddac`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00076125', 'lastCheck': '9.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00068848', 'lastCheck': '9.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:57:18,899::task::595::Storage.TaskManager.Task::(_updateState) Task=`304e7b85-ec83-414f-916b-7254e47dddac`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:57:18,899::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:57:18,899::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:57:18,899::task::993::Storage.TaskManager.Task::(_decref) Task=`304e7b85-ec83-414f-916b-7254e47dddac`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:57:18,911::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:57:19,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:57:19,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:57:19,139::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000840926 s, 441 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:57:19,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000613986 s, 852 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:20,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:20,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915429 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:22,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:23,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.35621 s, 755 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:57:24,070::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:57:24,071::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:57:25,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:26,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916424 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:28,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:28,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091372 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:57:29,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:57:29,163::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 14:57:29,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000765843 s, 484 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:57:29,185::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 14:57:29,186::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 14:57:29,189::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 14:57:29,191::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 14:57:29,191::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 14:57:29,191::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 14:57:29,192::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:57:29,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645556 s, 810 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:57:30,110::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:57:30,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:30,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546285 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:32,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:32,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914844 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:57:34,002::task::595::Storage.TaskManager.Task::(_updateState) Task=`afb6c996-3c57-4d44-82ac-e9d242eef146`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:57:34,003::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:57:34,003::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000765843', 'lastCheck': '4.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000645556', 'lastCheck': '4.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:57:34,003::task::1191::Storage.TaskManager.Task::(prepare) Task=`afb6c996-3c57-4d44-82ac-e9d242eef146`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000765843', 'lastCheck': '4.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000645556', 'lastCheck': '4.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:57:34,004::task::595::Storage.TaskManager.Task::(_updateState) Task=`afb6c996-3c57-4d44-82ac-e9d242eef146`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:57:34,004::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:57:34,004::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:57:34,004::task::993::Storage.TaskManager.Task::(_decref) Task=`afb6c996-3c57-4d44-82ac-e9d242eef146`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:57:34,016::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:57:34,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:57:34,077::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:57:34,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:35,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911112 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:37,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:37,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:39,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 14:57:39,240::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:57:39,252::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:57:39,265::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670633 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:57:39,268::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713918 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:39,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.128794 s, 8.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:41,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:41,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915978 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:43,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:43,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549859 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:57:44,084::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:57:44,086::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:57:45,123::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:57:45,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:46,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0971415 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:48,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:48,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:57:48,490::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-11142::DEBUG::2014-07-14 14:57:49,119::task::595::Storage.TaskManager.Task::(_updateState) Task=`87d39611-4479-4f56-b80c-ff94c4690653`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:57:49,119::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:57:49,119::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000670633', 'lastCheck': '9.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000713918', 'lastCheck': '9.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:57:49,119::task::1191::Storage.TaskManager.Task::(prepare) Task=`87d39611-4479-4f56-b80c-ff94c4690653`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000670633', 'lastCheck': '9.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000713918', 'lastCheck': '9.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:57:49,120::task::595::Storage.TaskManager.Task::(_updateState) Task=`87d39611-4479-4f56-b80c-ff94c4690653`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:57:49,120::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:57:49,120::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:57:49,120::task::993::Storage.TaskManager.Task::(_decref) Task=`87d39611-4479-4f56-b80c-ff94c4690653`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:57:49,132::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:57:49,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:57:49,749::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:57:49,761::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673421 s, 551 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:57:49,762::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000762221 s, 686 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:50,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:50,471::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938308 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:52,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:52,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913955 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:57:54,097::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:57:54,098::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:57:54,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:55,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.516646 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:57,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:57,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914331 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:57:59,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:57:59,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913546 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:57:59,766::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 14:57:59,783::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:57:59,798::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 14:57:59,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00392325 s, 133 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:57:59,800::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 14:57:59,803::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 14:57:59,805::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 14:57:59,805::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 14:57:59,805::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 14:57:59,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:57:59,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665869 s, 557 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:58:00,126::task::595::Storage.TaskManager.Task::(_updateState) Task=`502b2564-abe3-4aee-aa95-987931cac6f8`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:58:00,126::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:58:00,129::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:58:00,129::task::1191::Storage.TaskManager.Task::(prepare) Task=`502b2564-abe3-4aee-aa95-987931cac6f8`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:58:00,129::task::595::Storage.TaskManager.Task::(_updateState) Task=`502b2564-abe3-4aee-aa95-987931cac6f8`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:58:00,129::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:58:00,129::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:58:00,130::task::993::Storage.TaskManager.Task::(_decref) Task=`502b2564-abe3-4aee-aa95-987931cac6f8`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:58:00,155::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:58:01,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:02,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.3622 s, 752 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:58:04,109::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:58:04,119::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 14:58:04,225::task::595::Storage.TaskManager.Task::(_updateState) Task=`d03a63f3-e1b4-4a91-a367-6f07babf62cb`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:58:04,225::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:58:04,225::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000665869', 'lastCheck': '4.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00392325', 'lastCheck': '4.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:58:04,225::task::1191::Storage.TaskManager.Task::(prepare) Task=`d03a63f3-e1b4-4a91-a367-6f07babf62cb`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000665869', 'lastCheck': '4.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00392325', 'lastCheck': '4.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:58:04,226::task::595::Storage.TaskManager.Task::(_updateState) Task=`d03a63f3-e1b4-4a91-a367-6f07babf62cb`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:58:04,226::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:58:04,226::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:58:04,226::task::993::Storage.TaskManager.Task::(_decref) Task=`d03a63f3-e1b4-4a91-a367-6f07babf62cb`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:58:04,238::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:58:04,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:04,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091405 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:06,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:06,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915218 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:09,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:09,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916233 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:58:09,803::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:58:09,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660888 s, 791 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:58:09,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:58:09,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647132 s, 573 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:11,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:11,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913534 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:13,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:13,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091474 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:58:14,121::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:58:14,122::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:58:15,166::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:58:15,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:15,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913641 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:17,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:17,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913012 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:58:18,525::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-11142::DEBUG::2014-07-14 14:58:19,344::task::595::Storage.TaskManager.Task::(_updateState) Task=`ffa00266-b569-41a0-85be-cf6a97401b0c`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:58:19,344::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:58:19,345::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647132', 'lastCheck': '9.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000660888', 'lastCheck': '9.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:58:19,345::task::1191::Storage.TaskManager.Task::(prepare) Task=`ffa00266-b569-41a0-85be-cf6a97401b0c`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647132', 'lastCheck': '9.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000660888', 'lastCheck': '9.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:58:19,345::task::595::Storage.TaskManager.Task::(_updateState) Task=`ffa00266-b569-41a0-85be-cf6a97401b0c`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:58:19,345::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:58:19,345::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:58:19,345::task::993::Storage.TaskManager.Task::(_decref) Task=`ffa00266-b569-41a0-85be-cf6a97401b0c`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:58:19,357::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:58:19,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:19,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914917 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:58:19,819::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:58:19,830::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700705 s, 746 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:58:19,835::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:58:19,847::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000742655 s, 500 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:21,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:21,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914299 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:23,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:23,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913808 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:58:24,133::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:58:24,135::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:58:25,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:26,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.501595 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:28,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:28,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913905 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:58:29,834::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:58:29,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693965 s, 754 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:58:29,850::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:58:29,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674501 s, 550 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:58:30,178::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:58:30,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:30,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915922 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:32,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:32,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912805 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:58:34,143::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:58:34,144::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 14:58:34,447::task::595::Storage.TaskManager.Task::(_updateState) Task=`ba9e8be0-d7c3-4144-8e84-67327e88000c`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:58:34,447::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:58:34,447::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000674501', 'lastCheck': '4.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000693965', 'lastCheck': '4.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:58:34,447::task::1191::Storage.TaskManager.Task::(prepare) Task=`ba9e8be0-d7c3-4144-8e84-67327e88000c`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000674501', 'lastCheck': '4.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000693965', 'lastCheck': '4.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:58:34,448::task::595::Storage.TaskManager.Task::(_updateState) Task=`ba9e8be0-d7c3-4144-8e84-67327e88000c`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:58:34,448::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:58:34,448::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:58:34,448::task::993::Storage.TaskManager.Task::(_decref) Task=`ba9e8be0-d7c3-4144-8e84-67327e88000c`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:58:34,460::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:58:34,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:34,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913107 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:36,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:37,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914045 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:39,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:39,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091376 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:58:39,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:58:39,862::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00063577 s, 823 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:58:39,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:58:39,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642813 s, 577 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:41,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:41,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913588 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:43,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:43,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912716 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:58:44,155::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:58:44,156::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:58:45,190::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:58:45,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:45,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544365 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:48,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 14:58:48,557::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 14:58:49,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39936 s, 732 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 14:58:49,551::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3479875-91e1-42de-b0c6-4a8859ae66d2`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:58:49,551::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:58:49,551::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000642813', 'lastCheck': '9.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00063577', 'lastCheck': '9.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:58:49,552::task::1191::Storage.TaskManager.Task::(prepare) Task=`e3479875-91e1-42de-b0c6-4a8859ae66d2`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000642813', 'lastCheck': '9.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00063577', 'lastCheck': '9.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:58:49,552::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3479875-91e1-42de-b0c6-4a8859ae66d2`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:58:49,552::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:58:49,552::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:58:49,552::task::993::Storage.TaskManager.Task::(_decref) Task=`e3479875-91e1-42de-b0c6-4a8859ae66d2`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:58:49,563::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:58:49,867::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:58:49,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066299 s, 789 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:58:49,882::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:58:49,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660148 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:51,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:51,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091083 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:53,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 14:58:54,168::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:58:54,169::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:58:54,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36163 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:57,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:57,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917408 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:58:59,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:58:59,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913352 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:58:59,895::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:58:59,907::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:58:59,918::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00572203 s, 91.4 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:58:59,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000713102 s, 520 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:59:00,193::task::595::Storage.TaskManager.Task::(_updateState) Task=`082561b9-b04d-424a-a33f-5434b486061a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 14:59:00,193::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 14:59:00,196::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:59:00,196::task::1191::Storage.TaskManager.Task::(prepare) Task=`082561b9-b04d-424a-a33f-5434b486061a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 14:59:00,196::task::595::Storage.TaskManager.Task::(_updateState) Task=`082561b9-b04d-424a-a33f-5434b486061a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 14:59:00,197::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 14:59:00,197::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 14:59:00,197::task::993::Storage.TaskManager.Task::(_decref) Task=`082561b9-b04d-424a-a33f-5434b486061a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 14:59:00,223::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:59:01,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:01,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558665 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:03,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:03,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091366 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:59:04,172::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:59:04,174::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 14:59:04,667::task::595::Storage.TaskManager.Task::(_updateState) Task=`8513b4b2-8dfa-4c17-84c1-d8f73bdbdcfa`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:59:04,668::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:59:04,669::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000713102', 'lastCheck': '4.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00572203', 'lastCheck': '4.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:59:04,669::task::1191::Storage.TaskManager.Task::(prepare) Task=`8513b4b2-8dfa-4c17-84c1-d8f73bdbdcfa`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000713102', 'lastCheck': '4.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00572203', 'lastCheck': '4.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:59:04,669::task::595::Storage.TaskManager.Task::(_updateState) Task=`8513b4b2-8dfa-4c17-84c1-d8f73bdbdcfa`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:59:04,669::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:59:04,669::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:59:04,670::task::993::Storage.TaskManager.Task::(_decref) Task=`8513b4b2-8dfa-4c17-84c1-d8f73bdbdcfa`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:59:04,690::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:59:05,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:07,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.3622 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:09,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:09,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914859 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:59:09,923::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:59:09,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:59:09,950::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000591708 s, 884 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:59:09,952::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000735379 s, 505 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:11,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:11,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914499 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:13,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:13,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914314 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:59:14,181::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:59:14,182::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 14:59:15,234::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:59:15,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:15,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:17,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:18,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09138 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:59:18,589::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-11142::DEBUG::2014-07-14 14:59:19,783::task::595::Storage.TaskManager.Task::(_updateState) Task=`338b52ca-3f2f-47af-b48b-96bd53532461`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:59:19,783::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:59:19,783::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000735379', 'lastCheck': '9.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000591708', 'lastCheck': '9.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:59:19,783::task::1191::Storage.TaskManager.Task::(prepare) Task=`338b52ca-3f2f-47af-b48b-96bd53532461`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000735379', 'lastCheck': '9.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000591708', 'lastCheck': '9.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:59:19,784::task::595::Storage.TaskManager.Task::(_updateState) Task=`338b52ca-3f2f-47af-b48b-96bd53532461`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:59:19,784::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:59:19,784::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:59:19,784::task::993::Storage.TaskManager.Task::(_decref) Task=`338b52ca-3f2f-47af-b48b-96bd53532461`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:59:19,796::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 14:59:19,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:59:19,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:59:19,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628478 s, 590 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:59:19,984::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678527 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:20,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:20,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:22,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:22,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914059 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:59:24,190::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:59:24,191::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:59:24,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:24,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914404 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:26,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:26,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557508 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:28,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:29,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913169 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:59:29,990::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:59:30,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:59:30,012::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00082078 s, 452 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:59:30,014::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000754946 s, 693 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 14:59:30,246::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:59:31,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:31,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.521 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:33,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:34,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091169 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:59:34,202::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:59:34,203::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 14:59:34,875::task::595::Storage.TaskManager.Task::(_updateState) Task=`bac08a54-6309-40b7-bbfd-e9e88cd0f22d`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:59:34,876::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:59:34,876::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00082078', 'lastCheck': '4.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000754946', 'lastCheck': '4.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:59:34,876::task::1191::Storage.TaskManager.Task::(prepare) Task=`bac08a54-6309-40b7-bbfd-e9e88cd0f22d`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00082078', 'lastCheck': '4.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000754946', 'lastCheck': '4.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:59:34,876::task::595::Storage.TaskManager.Task::(_updateState) Task=`bac08a54-6309-40b7-bbfd-e9e88cd0f22d`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:59:34,876::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:59:34,876::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:59:34,877::task::993::Storage.TaskManager.Task::(_decref) Task=`bac08a54-6309-40b7-bbfd-e9e88cd0f22d`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:59:34,889::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:59:36,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:36,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914293 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:38,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:38,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914379 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 14:59:40,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:59:40,038::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:59:40,053::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00282354 s, 185 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:59:40,054::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000738034 s, 503 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:40,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:40,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543816 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:42,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:42,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914502 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:59:44,210::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:59:44,211::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:59:45,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 14:59:45,259::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 14:59:45,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559279 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:47,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:48,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541057 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 14:59:48,622::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-11142::DEBUG::2014-07-14 14:59:49,995::task::595::Storage.TaskManager.Task::(_updateState) Task=`c9faab72-70ba-4d8c-a5ba-75a20bcfa984`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 14:59:49,996::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 14:59:49,996::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000738034', 'lastCheck': '9.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00282354', 'lastCheck': '9.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:59:49,996::task::1191::Storage.TaskManager.Task::(prepare) Task=`c9faab72-70ba-4d8c-a5ba-75a20bcfa984`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000738034', 'lastCheck': '9.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00282354', 'lastCheck': '9.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 14:59:49,996::task::595::Storage.TaskManager.Task::(_updateState) Task=`c9faab72-70ba-4d8c-a5ba-75a20bcfa984`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 14:59:49,997::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 14:59:49,997::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 14:59:49,997::task::993::Storage.TaskManager.Task::(_decref) Task=`c9faab72-70ba-4d8c-a5ba-75a20bcfa984`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 14:59:50,010::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 14:59:50,063::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 14:59:50,063::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 14:59:50,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00198313 s, 264 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 14:59:50,085::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671345 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:50,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:50,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546436 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:52,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:53,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.505903 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 14:59:54,218::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 14:59:54,219::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 14:59:55,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:55,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913964 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:57,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:57,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913521 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 14:59:59,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 14:59:59,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912413 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:00:00,093::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:00:00,093::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:00:00,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689078 s, 538 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:00:00,117::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000822261 s, 636 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:00:00,262::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff9db32a-877c-4ce7-a53b-34dcb6f8e5c1`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:00:00,262::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:00:00,265::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:00:00,266::task::1191::Storage.TaskManager.Task::(prepare) Task=`ff9db32a-877c-4ce7-a53b-34dcb6f8e5c1`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:00:00,266::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff9db32a-877c-4ce7-a53b-34dcb6f8e5c1`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:00:00,266::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:00:00,266::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:00:00,266::task::993::Storage.TaskManager.Task::(_decref) Task=`ff9db32a-877c-4ce7-a53b-34dcb6f8e5c1`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:00:00,293::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:00:01,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:01,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913848 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:03,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:03,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914548 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:00:04,227::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:00:04,228::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 15:00:05,182::task::595::Storage.TaskManager.Task::(_updateState) Task=`4bd7a5dc-97e3-44e1-8b68-7a89743706df`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:00:05,183::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:00:05,183::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000689078', 'lastCheck': '5.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000822261', 'lastCheck': '5.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:00:05,183::task::1191::Storage.TaskManager.Task::(prepare) Task=`4bd7a5dc-97e3-44e1-8b68-7a89743706df`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000689078', 'lastCheck': '5.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000822261', 'lastCheck': '5.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:00:05,183::task::595::Storage.TaskManager.Task::(_updateState) Task=`4bd7a5dc-97e3-44e1-8b68-7a89743706df`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:00:05,183::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:00:05,184::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:00:05,184::task::993::Storage.TaskManager.Task::(_decref) Task=`4bd7a5dc-97e3-44e1-8b68-7a89743706df`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:00:05,195::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:00:05,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:06,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091447 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:08,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:08,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550771 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:00:10,119::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:00:10,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629374 s, 589 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:00:10,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:00:10,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621113 s, 842 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:10,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:12,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38852 s, 737 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:14,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:14,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914824 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:00:14,239::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:00:14,240::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:00:15,305::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:00:16,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:16,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544976 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:00:18,653::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:00:18,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:18,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912418 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:00:20,147::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:00:20,159::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:00:20,171::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639056 s, 581 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:00:20,173::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00119934 s, 436 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:00:20,290::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf59eec3-cb4d-4bbf-b923-78714946bcd9`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:00:20,290::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:00:20,290::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000639056', 'lastCheck': '0.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00119934', 'lastCheck': '0.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:00:20,290::task::1191::Storage.TaskManager.Task::(prepare) Task=`bf59eec3-cb4d-4bbf-b923-78714946bcd9`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000639056', 'lastCheck': '0.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00119934', 'lastCheck': '0.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:00:20,291::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf59eec3-cb4d-4bbf-b923-78714946bcd9`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:00:20,291::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:00:20,291::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:00:20,291::task::993::Storage.TaskManager.Task::(_decref) Task=`bf59eec3-cb4d-4bbf-b923-78714946bcd9`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:00:20,302::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:00:20,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:20,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912586 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:22,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:00:24,247::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:00:24,248::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:00:24,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38761 s, 738 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:26,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:26,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913273 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:28,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:28,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913515 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:00:30,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:00:30,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:00:30,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623949 s, 595 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:00:30,200::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000708491 s, 738 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:00:30,317::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:00:30,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:30,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912361 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:32,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:32,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915289 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:00:34,259::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:00:34,260::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:00:34,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:34,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914526 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:00:35,387::task::595::Storage.TaskManager.Task::(_updateState) Task=`0dde991c-e045-4972-a88e-569b758d617a`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:00:35,387::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:00:35,387::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000623949', 'lastCheck': '5.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000708491', 'lastCheck': '5.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:00:35,387::task::1191::Storage.TaskManager.Task::(prepare) Task=`0dde991c-e045-4972-a88e-569b758d617a`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000623949', 'lastCheck': '5.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000708491', 'lastCheck': '5.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:00:35,388::task::595::Storage.TaskManager.Task::(_updateState) Task=`0dde991c-e045-4972-a88e-569b758d617a`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:00:35,388::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:00:35,388::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:00:35,388::task::993::Storage.TaskManager.Task::(_decref) Task=`0dde991c-e045-4972-a88e-569b758d617a`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:00:35,400::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:00:36,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:37,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916025 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:39,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:39,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912989 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:00:40,204::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:00:40,215::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:00:40,231::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639811 s, 580 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:00:40,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00072432 s, 722 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:41,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:41,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.501785 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:43,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:00:44,272::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:00:44,273::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:00:45,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.35355 s, 757 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:00:45,330::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:00:47,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:47,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912941 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:00:48,685::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:00:49,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:49,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132667 s, 7.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:00:50,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:00:50,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:00:50,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648725 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:00:50,265::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000712633 s, 734 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:00:50,497::task::595::Storage.TaskManager.Task::(_updateState) Task=`a6cc52f8-839c-4c75-ba36-90bac40c460b`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:00:50,497::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:00:50,498::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648725', 'lastCheck': '0.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000712633', 'lastCheck': '0.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:00:50,498::task::1191::Storage.TaskManager.Task::(prepare) Task=`a6cc52f8-839c-4c75-ba36-90bac40c460b`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648725', 'lastCheck': '0.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000712633', 'lastCheck': '0.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:00:50,498::task::595::Storage.TaskManager.Task::(_updateState) Task=`a6cc52f8-839c-4c75-ba36-90bac40c460b`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:00:50,498::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:00:50,498::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:00:50,498::task::993::Storage.TaskManager.Task::(_decref) Task=`a6cc52f8-839c-4c75-ba36-90bac40c460b`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:00:50,510::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:00:51,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:51,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915116 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:53,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:53,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913814 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:00:54,281::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:00:54,282::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:00:55,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:55,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0951628 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:00:57,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:00:58,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.516601 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:01:00,268::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:01:00,281::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000733062 s, 506 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:01:00,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:01:00,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629663 s, 831 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:00,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:01:00,334::task::595::Storage.TaskManager.Task::(_updateState) Task=`325175fb-02fe-4466-b912-294177dbbcf2`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:01:00,334::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:01:00,419::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:01:00,420::task::1191::Storage.TaskManager.Task::(prepare) Task=`325175fb-02fe-4466-b912-294177dbbcf2`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:01:00,420::task::595::Storage.TaskManager.Task::(_updateState) Task=`325175fb-02fe-4466-b912-294177dbbcf2`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:01:00,421::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:01:00,421::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:01:00,422::task::993::Storage.TaskManager.Task::(_decref) Task=`325175fb-02fe-4466-b912-294177dbbcf2`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 15:01:00,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0962321 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:01:00,451::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:01:02,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:02,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913779 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:01:04,292::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:01:04,294::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:04,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:04,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922073 s, 11.1 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:01:05,596::task::595::Storage.TaskManager.Task::(_updateState) Task=`d078b9c4-fecc-493e-b1a4-7da95b4fdf95`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:01:05,596::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:01:05,597::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000733062', 'lastCheck': '5.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000629663', 'lastCheck': '5.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:01:05,597::task::1191::Storage.TaskManager.Task::(prepare) Task=`d078b9c4-fecc-493e-b1a4-7da95b4fdf95`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000733062', 'lastCheck': '5.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000629663', 'lastCheck': '5.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:01:05,597::task::595::Storage.TaskManager.Task::(_updateState) Task=`d078b9c4-fecc-493e-b1a4-7da95b4fdf95`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:01:05,597::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:01:05,597::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:01:05,597::task::993::Storage.TaskManager.Task::(_decref) Task=`d078b9c4-fecc-493e-b1a4-7da95b4fdf95`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:01:05,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:06,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:06,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914184 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:08,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:08,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913893 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:01:10,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:01:10,308::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000820276 s, 452 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:01:10,308::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:01:10,321::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000814429 s, 642 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:10,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:11,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913709 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:13,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:13,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912548 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:01:14,301::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:01:14,302::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:15,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:15,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916028 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:01:15,466::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:01:17,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:17,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912938 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:01:18,717::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:01:19,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:19,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543984 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:01:20,322::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:01:20,335::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000601344 s, 617 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:01:20,336::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:01:20,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000762194 s, 686 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:01:20,704::task::595::Storage.TaskManager.Task::(_updateState) Task=`b657dc36-b0be-4354-8e38-3ef462ef850f`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:01:20,704::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:01:20,704::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000601344', 'lastCheck': '0.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000762194', 'lastCheck': '0.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:01:20,704::task::1191::Storage.TaskManager.Task::(prepare) Task=`b657dc36-b0be-4354-8e38-3ef462ef850f`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000601344', 'lastCheck': '0.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000762194', 'lastCheck': '0.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:01:20,704::task::595::Storage.TaskManager.Task::(_updateState) Task=`b657dc36-b0be-4354-8e38-3ef462ef850f`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:01:20,705::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:01:20,705::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:01:20,705::task::993::Storage.TaskManager.Task::(_decref) Task=`b657dc36-b0be-4354-8e38-3ef462ef850f`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:01:20,716::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:21,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:22,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912897 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:24,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:24,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913609 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:01:24,311::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:01:24,312::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:26,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:26,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091414 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:28,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:28,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091358 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:01:30,350::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:01:30,362::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:01:30,372::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676927 s, 548 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:01:30,374::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000775307 s, 675 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:30,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:01:30,479::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:01:30,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913236 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:32,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:32,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912637 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:01:34,321::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:01:34,322::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:34,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:34,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913855 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:01:35,804::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c56c30a-b392-455e-a4bb-43bed1a6f062`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:01:35,805::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:01:35,805::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000676927', 'lastCheck': '5.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000775307', 'lastCheck': '5.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:01:35,805::task::1191::Storage.TaskManager.Task::(prepare) Task=`0c56c30a-b392-455e-a4bb-43bed1a6f062`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000676927', 'lastCheck': '5.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000775307', 'lastCheck': '5.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:01:35,805::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c56c30a-b392-455e-a4bb-43bed1a6f062`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:01:35,805::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:01:35,805::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:01:35,806::task::993::Storage.TaskManager.Task::(_decref) Task=`0c56c30a-b392-455e-a4bb-43bed1a6f062`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:01:35,817::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:36,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:36,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914726 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:38,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:38,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913783 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:01:40,377::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:01:40,388::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:01:40,399::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063022 s, 589 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:01:40,402::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641363 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:41,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:41,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918343 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:43,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:43,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909461 s, 11.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:01:44,331::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:01:44,333::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:45,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:45,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913487 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:01:45,492::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:01:47,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:47,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913376 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:01:48,749::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:01:49,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:50,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.516306 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:01:50,406::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:01:50,417::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:01:50,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691132 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:01:50,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658957 s, 794 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:01:50,896::task::595::Storage.TaskManager.Task::(_updateState) Task=`ca197d4b-f62d-4928-a18d-8f5b935f9391`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:01:50,896::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:01:50,897::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000691132', 'lastCheck': '0.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000658957', 'lastCheck': '0.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:01:50,897::task::1191::Storage.TaskManager.Task::(prepare) Task=`ca197d4b-f62d-4928-a18d-8f5b935f9391`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000691132', 'lastCheck': '0.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000658957', 'lastCheck': '0.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:01:50,897::task::595::Storage.TaskManager.Task::(_updateState) Task=`ca197d4b-f62d-4928-a18d-8f5b935f9391`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:01:50,898::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:01:50,898::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:01:50,898::task::993::Storage.TaskManager.Task::(_decref) Task=`ca197d4b-f62d-4928-a18d-8f5b935f9391`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:01:50,910::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:52,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:52,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914706 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:54,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:54,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914172 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:01:54,343::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:01:54,344::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:01:56,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:56,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559058 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:01:58,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:01:58,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912719 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:02:00,437::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:02:00,449::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:02:00,459::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000809022 s, 459 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:02:00,462::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694596 s, 753 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:02:00,495::task::595::Storage.TaskManager.Task::(_updateState) Task=`d426f7ca-3035-4d72-ac2e-7980a7320dcc`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:02:00,496::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:02:00,498::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:02:00,498::task::1191::Storage.TaskManager.Task::(prepare) Task=`d426f7ca-3035-4d72-ac2e-7980a7320dcc`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:02:00,498::task::595::Storage.TaskManager.Task::(_updateState) Task=`d426f7ca-3035-4d72-ac2e-7980a7320dcc`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:02:00,498::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:02:00,498::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:02:00,498::task::993::Storage.TaskManager.Task::(_decref) Task=`d426f7ca-3035-4d72-ac2e-7980a7320dcc`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:02:00,527::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:02:00,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:01,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917646 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:03,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:03,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914416 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:02:04,355::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:02:04,357::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:02:05,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:05,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913312 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:02:05,992::task::595::Storage.TaskManager.Task::(_updateState) Task=`eeaaa042-95ab-459b-91f2-c91b72bfb0af`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:02:05,992::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:02:05,992::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000809022', 'lastCheck': '5.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000694596', 'lastCheck': '5.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:02:05,993::task::1191::Storage.TaskManager.Task::(prepare) Task=`eeaaa042-95ab-459b-91f2-c91b72bfb0af`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000809022', 'lastCheck': '5.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000694596', 'lastCheck': '5.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:02:05,993::task::595::Storage.TaskManager.Task::(_updateState) Task=`eeaaa042-95ab-459b-91f2-c91b72bfb0af`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:02:05,993::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:02:05,993::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:02:05,993::task::993::Storage.TaskManager.Task::(_decref) Task=`eeaaa042-95ab-459b-91f2-c91b72bfb0af`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:02:06,005::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:02:07,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:07,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915101 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:09,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:09,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091332 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:02:10,463::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:02:10,476::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000760134 s, 488 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:02:10,477::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:02:10,489::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000805554 s, 649 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:11,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:11,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091438 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:13,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:14,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540969 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:02:14,364::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:02:14,366::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:02:15,540::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:02:16,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:16,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0959784 s, 10.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:18,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:02:18,782::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:02:19,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556923 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:02:20,491::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:02:20,503::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000773436 s, 480 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:02:20,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:02:20,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000741725 s, 705 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:21,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:02:21,090::task::595::Storage.TaskManager.Task::(_updateState) Task=`089ba39b-9ca1-4dfa-bb2d-a24046eba920`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:02:21,091::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:02:21,091::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000773436', 'lastCheck': '0.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000741725', 'lastCheck': '0.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:02:21,091::task::1191::Storage.TaskManager.Task::(prepare) Task=`089ba39b-9ca1-4dfa-bb2d-a24046eba920`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000773436', 'lastCheck': '0.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000741725', 'lastCheck': '0.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:02:21,091::task::595::Storage.TaskManager.Task::(_updateState) Task=`089ba39b-9ca1-4dfa-bb2d-a24046eba920`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:02:21,092::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:02:21,092::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:02:21,092::task::993::Storage.TaskManager.Task::(_decref) Task=`089ba39b-9ca1-4dfa-bb2d-a24046eba920`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:02:21,107::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:02:21,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916356 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:23,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:23,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.408058 s, 2.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:02:24,376::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:02:24,378::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:02:25,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:25,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913949 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:27,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:27,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914772 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:29,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:29,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912886 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:02:30,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:02:30,530::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:02:30,535::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667426 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:02:30,550::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:02:30,552::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:02:30,555::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-113::DEBUG::2014-07-14 15:02:30,557::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 15:02:30,557::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:02:30,558::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:02:30,558::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:02:30,560::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:02:30,571::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680855 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:31,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:32,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914319 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:34,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:02:34,387::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:02:34,388::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:02:34,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.508458 s, 2.0 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:02:36,187::task::595::Storage.TaskManager.Task::(_updateState) Task=`0eea4360-f3bd-4c6a-ab1a-f94f66a22857`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:02:36,187::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:02:36,187::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667426', 'lastCheck': '5.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000680855', 'lastCheck': '5.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:02:36,187::task::1191::Storage.TaskManager.Task::(prepare) Task=`0eea4360-f3bd-4c6a-ab1a-f94f66a22857`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667426', 'lastCheck': '5.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000680855', 'lastCheck': '5.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:02:36,187::task::595::Storage.TaskManager.Task::(_updateState) Task=`0eea4360-f3bd-4c6a-ab1a-f94f66a22857`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:02:36,188::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:02:36,188::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:02:36,188::task::993::Storage.TaskManager.Task::(_decref) Task=`0eea4360-f3bd-4c6a-ab1a-f94f66a22857`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:02:36,199::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:02:36,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:36,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913298 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:38,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:38,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915404 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:02:40,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:02:40,623::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:02:40,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00448569 s, 117 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:02:40,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000607573 s, 611 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:40,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:40,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.133085 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:42,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:43,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914357 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:02:44,396::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:02:44,398::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:02:45,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:45,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913754 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:02:45,573::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:02:47,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:47,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914166 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:02:48,819::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:02:49,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:49,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912703 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:02:50,652::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:02:50,663::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:02:50,674::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000825702 s, 449 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:02:50,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000930473 s, 562 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:02:51,288::task::595::Storage.TaskManager.Task::(_updateState) Task=`a7b8463b-4a98-4ffc-af72-df9c63e0a637`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:02:51,288::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:02:51,289::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000825702', 'lastCheck': '0.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000930473', 'lastCheck': '0.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:02:51,289::task::1191::Storage.TaskManager.Task::(prepare) Task=`a7b8463b-4a98-4ffc-af72-df9c63e0a637`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000825702', 'lastCheck': '0.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000930473', 'lastCheck': '0.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:02:51,289::task::595::Storage.TaskManager.Task::(_updateState) Task=`a7b8463b-4a98-4ffc-af72-df9c63e0a637`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:02:51,289::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:02:51,289::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:02:51,290::task::993::Storage.TaskManager.Task::(_decref) Task=`a7b8463b-4a98-4ffc-af72-df9c63e0a637`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:02:51,301::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:02:51,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:51,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925298 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:53,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:53,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913735 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:02:54,408::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:02:54,409::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:02:55,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:55,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091164 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:57,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:02:57,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912484 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:02:59,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:00,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091198 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:03:00,577::task::595::Storage.TaskManager.Task::(_updateState) Task=`d063dc60-56ec-4b2a-bf99-65cb4a2b6e8e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:03:00,577::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:03:00,580::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:03:00,580::task::1191::Storage.TaskManager.Task::(prepare) Task=`d063dc60-56ec-4b2a-bf99-65cb4a2b6e8e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:03:00,581::task::595::Storage.TaskManager.Task::(_updateState) Task=`d063dc60-56ec-4b2a-bf99-65cb4a2b6e8e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:03:00,581::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:03:00,581::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:03:00,581::task::993::Storage.TaskManager.Task::(_decref) Task=`d063dc60-56ec-4b2a-bf99-65cb4a2b6e8e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:03:00,608::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:03:00,677::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 15:03:00,685::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:03:00,700::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000845034 s, 619 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:03:00,710::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:03:00,711::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:03:00,714::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:03:00,716::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:03:00,716::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:03:00,716::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:03:00,717::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:03:00,729::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000743809 s, 499 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:02,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:03,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36325 s, 751 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:03:04,416::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:03:04,417::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:03:05,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:05,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912842 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:03:06,392::task::595::Storage.TaskManager.Task::(_updateState) Task=`ac9bfc27-cf9e-462f-a56b-06f6d6feb8c1`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:03:06,392::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:03:06,392::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000743809', 'lastCheck': '5.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000845034', 'lastCheck': '5.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:03:06,393::task::1191::Storage.TaskManager.Task::(prepare) Task=`ac9bfc27-cf9e-462f-a56b-06f6d6feb8c1`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000743809', 'lastCheck': '5.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000845034', 'lastCheck': '5.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:03:06,393::task::595::Storage.TaskManager.Task::(_updateState) Task=`ac9bfc27-cf9e-462f-a56b-06f6d6feb8c1`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:03:06,393::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:03:06,393::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:03:06,393::task::993::Storage.TaskManager.Task::(_decref) Task=`ac9bfc27-cf9e-462f-a56b-06f6d6feb8c1`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:03:06,405::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:03:07,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:07,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913509 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:09,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:09,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910868 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:03:10,706::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:03:10,719::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682558 s, 766 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:03:10,732::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:03:10,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662671 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:11,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:11,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913122 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:13,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:14,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913513 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:03:14,428::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:03:14,429::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:03:15,626::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:03:16,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:16,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967642 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:18,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:18,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922017 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:03:18,851::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:03:20,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:03:20,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:03:20,912::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:03:20,928::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656367 s, 797 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:03:20,931::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673694 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:20,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547914 s, 1.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:03:21,509::task::595::Storage.TaskManager.Task::(_updateState) Task=`18ae8865-7e77-45dc-a3f1-3d43561d9175`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:03:21,509::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:03:21,509::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000673694', 'lastCheck': '0.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656367', 'lastCheck': '0.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:03:21,510::task::1191::Storage.TaskManager.Task::(prepare) Task=`18ae8865-7e77-45dc-a3f1-3d43561d9175`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000673694', 'lastCheck': '0.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656367', 'lastCheck': '0.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:03:21,510::task::595::Storage.TaskManager.Task::(_updateState) Task=`18ae8865-7e77-45dc-a3f1-3d43561d9175`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:03:21,510::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:03:21,510::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:03:21,510::task::993::Storage.TaskManager.Task::(_decref) Task=`18ae8865-7e77-45dc-a3f1-3d43561d9175`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:03:21,521::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:03:22,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:23,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549605 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:03:24,440::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:03:24,442::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:03:25,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:25,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923786 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:27,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:27,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914483 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:29,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:29,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913332 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:03:30,639::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 15:03:30,937::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:03:30,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:03:30,959::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700564 s, 747 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:03:30,960::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000701915 s, 529 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:31,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:32,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545804 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:03:34,452::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:03:34,453::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:03:34,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:34,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913499 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:36,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:03:36,611::task::595::Storage.TaskManager.Task::(_updateState) Task=`920575f6-fda2-4a1e-80c9-97ba7c0cbf2d`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:03:36,611::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:03:36,611::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000701915', 'lastCheck': '5.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000700564', 'lastCheck': '5.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:03:36,612::task::1191::Storage.TaskManager.Task::(prepare) Task=`920575f6-fda2-4a1e-80c9-97ba7c0cbf2d`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000701915', 'lastCheck': '5.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000700564', 'lastCheck': '5.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:03:36,612::task::595::Storage.TaskManager.Task::(_updateState) Task=`920575f6-fda2-4a1e-80c9-97ba7c0cbf2d`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:03:36,612::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:03:36,612::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:03:36,612::task::993::Storage.TaskManager.Task::(_decref) Task=`920575f6-fda2-4a1e-80c9-97ba7c0cbf2d`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:03:36,627::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:03:37,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549559 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:39,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:39,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913632 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:03:40,967::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:03:40,978::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:03:40,989::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065635 s, 797 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:03:40,992::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000749602 s, 495 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:41,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:41,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915276 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:43,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:43,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09127 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:03:44,460::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:03:44,461::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:03:45,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:45,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0969928 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:03:45,654::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:03:47,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:47,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913315 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:03:48,882::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:03:49,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:49,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913864 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:03:50,997::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:03:51,007::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:03:51,019::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000624433 s, 838 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:03:51,020::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704883 s, 526 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:03:51,719::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c73fb94-6613-426e-80f7-a8a0a337f73e`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:03:51,719::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:03:51,719::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000704883', 'lastCheck': '0.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000624433', 'lastCheck': '0.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:03:51,720::task::1191::Storage.TaskManager.Task::(prepare) Task=`1c73fb94-6613-426e-80f7-a8a0a337f73e`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000704883', 'lastCheck': '0.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000624433', 'lastCheck': '0.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:03:51,720::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c73fb94-6613-426e-80f7-a8a0a337f73e`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:03:51,720::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:03:51,720::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:03:51,720::task::993::Storage.TaskManager.Task::(_decref) Task=`1c73fb94-6613-426e-80f7-a8a0a337f73e`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:03:51,732::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:03:51,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:52,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914194 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:54,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:54,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912776 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:03:54,470::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:03:54,471::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:03:56,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:56,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542912 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:03:58,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:03:58,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914413 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:04:00,659::task::595::Storage.TaskManager.Task::(_updateState) Task=`30adcc87-909a-43f7-bc80-e897f7cd6264`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:04:00,659::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:04:00,662::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:04:00,662::task::1191::Storage.TaskManager.Task::(prepare) Task=`30adcc87-909a-43f7-bc80-e897f7cd6264`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:04:00,663::task::595::Storage.TaskManager.Task::(_updateState) Task=`30adcc87-909a-43f7-bc80-e897f7cd6264`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:04:00,663::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:04:00,663::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:04:00,663::task::993::Storage.TaskManager.Task::(_decref) Task=`30adcc87-909a-43f7-bc80-e897f7cd6264`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:04:00,691::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:04:00,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:00,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913148 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:04:01,027::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:04:01,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:04:01,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00509772 s, 103 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:04:01,051::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000711736 s, 521 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:03,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:03,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913151 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:04:04,479::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:04:04,481::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:04:05,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:05,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913664 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:04:06,845::task::595::Storage.TaskManager.Task::(_updateState) Task=`e1838a9f-61b1-449f-994f-9923c6abfd79`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:04:06,846::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:04:06,846::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000711736', 'lastCheck': '5.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00509772', 'lastCheck': '5.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:04:06,846::task::1191::Storage.TaskManager.Task::(prepare) Task=`e1838a9f-61b1-449f-994f-9923c6abfd79`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000711736', 'lastCheck': '5.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00509772', 'lastCheck': '5.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:04:06,846::task::595::Storage.TaskManager.Task::(_updateState) Task=`e1838a9f-61b1-449f-994f-9923c6abfd79`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:04:06,846::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:04:06,847::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:04:06,847::task::993::Storage.TaskManager.Task::(_decref) Task=`e1838a9f-61b1-449f-994f-9923c6abfd79`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:04:06,859::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:04:07,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:07,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913636 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:09,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:09,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54352 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:04:11,057::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:04:11,069::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:04:11,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652256 s, 802 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:04:11,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644025 s, 576 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:11,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:12,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913563 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:14,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:14,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914367 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:04:14,485::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:04:14,486::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:04:15,707::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:04:16,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:16,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557876 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:18,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:04:18,914::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:04:19,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.537163 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:04:21,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:04:21,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:04:21,116::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677532 s, 772 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:04:21,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000732633 s, 506 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:21,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:21,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919564 s, 11.1 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:04:21,963::task::595::Storage.TaskManager.Task::(_updateState) Task=`a85e5ee6-f678-4b92-8211-6e6db2597229`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:04:21,963::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:04:21,963::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000732633', 'lastCheck': '0.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000677532', 'lastCheck': '0.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:04:21,964::task::1191::Storage.TaskManager.Task::(prepare) Task=`a85e5ee6-f678-4b92-8211-6e6db2597229`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000732633', 'lastCheck': '0.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000677532', 'lastCheck': '0.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:04:21,964::task::595::Storage.TaskManager.Task::(_updateState) Task=`a85e5ee6-f678-4b92-8211-6e6db2597229`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:04:21,964::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:04:21,964::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:04:21,964::task::993::Storage.TaskManager.Task::(_decref) Task=`a85e5ee6-f678-4b92-8211-6e6db2597229`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:04:21,975::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:04:23,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:23,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913036 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:04:24,492::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:04:24,493::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:04:25,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:26,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554069 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:28,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:28,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915902 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:30,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:30,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914188 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:04:30,721::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 15:04:31,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:04:31,133::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:04:31,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629537 s, 831 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:04:31,147::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00060526 s, 613 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:32,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:32,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914421 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:34,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:04:34,504::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:04:34,505::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:04:34,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916183 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:36,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:36,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913544 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:04:37,084::task::595::Storage.TaskManager.Task::(_updateState) Task=`09c5ba2a-3d9d-4def-9a87-cd90f0fe0593`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:04:37,084::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:04:37,085::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00060526', 'lastCheck': '5.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000629537', 'lastCheck': '5.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:04:37,085::task::1191::Storage.TaskManager.Task::(prepare) Task=`09c5ba2a-3d9d-4def-9a87-cd90f0fe0593`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00060526', 'lastCheck': '5.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000629537', 'lastCheck': '5.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:04:37,085::task::595::Storage.TaskManager.Task::(_updateState) Task=`09c5ba2a-3d9d-4def-9a87-cd90f0fe0593`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:04:37,085::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:04:37,085::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:04:37,086::task::993::Storage.TaskManager.Task::(_decref) Task=`09c5ba2a-3d9d-4def-9a87-cd90f0fe0593`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:04:37,097::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:04:38,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:38,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912738 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:40,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:40,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914706 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:04:41,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:04:41,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:04:41,179::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064142 s, 815 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:04:41,181::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000870737 s, 426 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:42,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:43,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913332 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:04:44,512::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:04:44,513::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:04:45,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:45,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.298204 s, 3.4 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:04:45,743::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:04:47,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:47,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544624 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:04:48,948::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:04:49,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:50,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556866 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:04:51,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:04:51,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:04:51,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647204 s, 808 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:04:51,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647532 s, 573 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:04:52,188::task::595::Storage.TaskManager.Task::(_updateState) Task=`a429755d-7810-476b-a28e-ba66d32bf63c`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:04:52,188::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:04:52,188::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647532', 'lastCheck': '1.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647204', 'lastCheck': '1.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:04:52,189::task::1191::Storage.TaskManager.Task::(prepare) Task=`a429755d-7810-476b-a28e-ba66d32bf63c`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647532', 'lastCheck': '1.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647204', 'lastCheck': '1.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:04:52,189::task::595::Storage.TaskManager.Task::(_updateState) Task=`a429755d-7810-476b-a28e-ba66d32bf63c`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:04:52,189::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:04:52,189::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:04:52,189::task::993::Storage.TaskManager.Task::(_decref) Task=`a429755d-7810-476b-a28e-ba66d32bf63c`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:04:52,201::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:04:52,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:53,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546685 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:04:54,519::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:04:54,521::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:04:55,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:55,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912946 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:57,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:57,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912909 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:04:59,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:04:59,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912849 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:05:00,747::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a3ea639-1650-43c3-93c3-3431eeb0b290`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:05:00,747::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:05:00,750::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:05:00,751::task::1191::Storage.TaskManager.Task::(prepare) Task=`4a3ea639-1650-43c3-93c3-3431eeb0b290`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:05:00,751::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a3ea639-1650-43c3-93c3-3431eeb0b290`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:05:00,751::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:05:00,751::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:05:00,751::task::993::Storage.TaskManager.Task::(_decref) Task=`4a3ea639-1650-43c3-93c3-3431eeb0b290`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:05:00,779::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 15:05:01,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:05:01,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:05:01,243::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661779 s, 790 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:05:01,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000792998 s, 468 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:01,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:01,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912872 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:03,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:03,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913902 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:05:04,528::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:05:04,529::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:05,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:05,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914334 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:05:07,291::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb9278df-81b5-46d6-ba63-fd0b83bcf2b0`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:05:07,291::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:05:07,291::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000792998', 'lastCheck': '6.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000661779', 'lastCheck': '6.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:05:07,292::task::1191::Storage.TaskManager.Task::(prepare) Task=`fb9278df-81b5-46d6-ba63-fd0b83bcf2b0`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000792998', 'lastCheck': '6.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000661779', 'lastCheck': '6.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:05:07,292::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb9278df-81b5-46d6-ba63-fd0b83bcf2b0`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:05:07,292::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:05:07,292::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:05:07,292::task::993::Storage.TaskManager.Task::(_decref) Task=`fb9278df-81b5-46d6-ba63-fd0b83bcf2b0`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:05:07,304::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:07,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:07,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912187 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:09,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:10,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54542 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:05:11,249::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:05:11,260::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:05:11,272::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00365156 s, 143 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:05:11,273::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664397 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:12,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:12,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926916 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:05:14,539::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:05:14,541::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:14,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:14,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912066 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:05:15,796::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:05:16,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:16,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914618 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:18,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:18,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913407 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:05:18,981::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:05:20,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:21,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913789 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:05:21,280::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:05:21,291::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:05:21,303::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00522724 s, 71.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:05:21,305::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000618115 s, 846 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:05:22,394::task::595::Storage.TaskManager.Task::(_updateState) Task=`e526562c-a964-4e7f-8d8d-e9d387b0499b`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:05:22,394::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:05:22,394::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00522724', 'lastCheck': '1.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000618115', 'lastCheck': '1.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:05:22,394::task::1191::Storage.TaskManager.Task::(prepare) Task=`e526562c-a964-4e7f-8d8d-e9d387b0499b`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00522724', 'lastCheck': '1.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000618115', 'lastCheck': '1.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:05:22,395::task::595::Storage.TaskManager.Task::(_updateState) Task=`e526562c-a964-4e7f-8d8d-e9d387b0499b`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:05:22,395::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:05:22,395::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:05:22,395::task::993::Storage.TaskManager.Task::(_decref) Task=`e526562c-a964-4e7f-8d8d-e9d387b0499b`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:05:22,406::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:23,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:23,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912987 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:05:24,548::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:05:24,549::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:25,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:25,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913154 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:27,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:27,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550274 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:29,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:05:30,809::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:05:31,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911006 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:05:31,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:05:31,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:05:31,319::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000833109 s, 445 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:05:31,332::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640389 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:33,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:33,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912882 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:05:34,560::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:05:34,561::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:35,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:35,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545239 s, 1.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:05:37,514::task::595::Storage.TaskManager.Task::(_updateState) Task=`47c297a7-bcb5-4f86-8dd9-2fa9cb67f788`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:05:37,514::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:05:37,515::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000833109', 'lastCheck': '6.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000640389', 'lastCheck': '6.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:05:37,515::task::1191::Storage.TaskManager.Task::(prepare) Task=`47c297a7-bcb5-4f86-8dd9-2fa9cb67f788`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000833109', 'lastCheck': '6.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000640389', 'lastCheck': '6.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:05:37,515::task::595::Storage.TaskManager.Task::(_updateState) Task=`47c297a7-bcb5-4f86-8dd9-2fa9cb67f788`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:05:37,515::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:05:37,515::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:05:37,515::task::993::Storage.TaskManager.Task::(_decref) Task=`47c297a7-bcb5-4f86-8dd9-2fa9cb67f788`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:05:37,526::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:37,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:37,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915904 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:39,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:40,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560109 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:05:41,336::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:05:41,347::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:05:41,358::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627938 s, 591 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:05:41,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00061718 s, 847 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:42,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:42,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913689 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:05:44,567::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:05:44,568::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:44,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:44,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914008 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:05:45,833::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:05:46,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:48,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38014 s, 742 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:05:49,015::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:05:50,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:50,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913366 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:05:51,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:05:51,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:05:51,387::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00568505 s, 65.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:05:51,388::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660758 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:52,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:52,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912922 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:05:52,614::task::595::Storage.TaskManager.Task::(_updateState) Task=`5af71ade-c836-442c-9a5c-dc71e487de83`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:05:52,615::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:05:52,615::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00568505', 'lastCheck': '1.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000660758', 'lastCheck': '1.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:05:52,615::task::1191::Storage.TaskManager.Task::(prepare) Task=`5af71ade-c836-442c-9a5c-dc71e487de83`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00568505', 'lastCheck': '1.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000660758', 'lastCheck': '1.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:05:52,615::task::595::Storage.TaskManager.Task::(_updateState) Task=`5af71ade-c836-442c-9a5c-dc71e487de83`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:05:52,615::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:05:52,616::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:05:52,616::task::993::Storage.TaskManager.Task::(_decref) Task=`5af71ade-c836-442c-9a5c-dc71e487de83`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:05:52,627::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:54,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:54,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913577 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:05:54,579::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:05:54,580::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:05:56,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:56,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914625 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:05:58,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:05:58,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912495 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:00,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:06:00,837::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc692d62-3b47-423d-bbef-7bb227a91a41`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:06:00,838::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:06:00,890::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:06:00,892::task::1191::Storage.TaskManager.Task::(prepare) Task=`dc692d62-3b47-423d-bbef-7bb227a91a41`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:06:00,892::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc692d62-3b47-423d-bbef-7bb227a91a41`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:06:00,893::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:06:00,894::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:06:00,895::task::993::Storage.TaskManager.Task::(_decref) Task=`dc692d62-3b47-423d-bbef-7bb227a91a41`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 15:06:00,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.096965 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:06:00,927::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:06:01,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:06:01,404::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:06:01,419::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0033784 s, 110 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:06:01,422::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668725 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:02,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:03,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913872 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:06:04,591::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:06:04,593::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:06:05,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:05,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913564 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:07,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:07,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913833 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:06:07,713::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2a2838a-1e87-4fa1-9f71-943635b7b5b0`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:06:07,713::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:06:07,714::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0033784', 'lastCheck': '6.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000668725', 'lastCheck': '6.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:06:07,714::task::1191::Storage.TaskManager.Task::(prepare) Task=`c2a2838a-1e87-4fa1-9f71-943635b7b5b0`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0033784', 'lastCheck': '6.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000668725', 'lastCheck': '6.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:06:07,714::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2a2838a-1e87-4fa1-9f71-943635b7b5b0`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:06:07,714::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:06:07,714::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:06:07,715::task::993::Storage.TaskManager.Task::(_decref) Task=`c2a2838a-1e87-4fa1-9f71-943635b7b5b0`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:06:07,730::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:06:09,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:09,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911929 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:11,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:06:11,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:06:11,501::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:06:11,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677649 s, 772 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:06:11,516::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663261 s, 559 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:11,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129221 s, 7.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:13,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:14,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:06:14,604::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:06:14,605::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:06:15,943::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:06:16,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:16,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0972174 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:18,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:18,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544704 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:06:19,047::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:06:20,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:21,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545955 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:06:21,526::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:06:21,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650915 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:06:21,539::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:06:21,553::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000778697 s, 672 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:06:22,842::task::595::Storage.TaskManager.Task::(_updateState) Task=`80dad1d9-6f0f-4c7a-8039-92f7ed3c7394`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:06:22,843::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:06:22,843::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650915', 'lastCheck': '1.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000778697', 'lastCheck': '1.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:06:22,843::task::1191::Storage.TaskManager.Task::(prepare) Task=`80dad1d9-6f0f-4c7a-8039-92f7ed3c7394`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650915', 'lastCheck': '1.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000778697', 'lastCheck': '1.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:06:22,843::task::595::Storage.TaskManager.Task::(_updateState) Task=`80dad1d9-6f0f-4c7a-8039-92f7ed3c7394`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:06:22,843::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:06:22,844::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:06:22,844::task::993::Storage.TaskManager.Task::(_decref) Task=`80dad1d9-6f0f-4c7a-8039-92f7ed3c7394`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:06:22,860::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:06:23,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:24,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542895 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:06:24,608::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:06:24,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:06:26,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:26,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911995 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:28,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:28,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913546 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:30,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:30,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55758 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:06:30,957::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:06:31,554::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:06:31,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000666501 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:06:31,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:06:31,581::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000918199 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:32,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:32,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102569 s, 10.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:06:34,620::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:06:34,621::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:06:35,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:35,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:37,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:37,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912653 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:06:37,945::task::595::Storage.TaskManager.Task::(_updateState) Task=`faec10e8-5c66-4905-9f50-b5081cb719ef`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:06:37,945::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:06:37,946::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000666501', 'lastCheck': '6.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000918199', 'lastCheck': '6.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:06:37,946::task::1191::Storage.TaskManager.Task::(prepare) Task=`faec10e8-5c66-4905-9f50-b5081cb719ef`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000666501', 'lastCheck': '6.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000918199', 'lastCheck': '6.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:06:37,946::task::595::Storage.TaskManager.Task::(_updateState) Task=`faec10e8-5c66-4905-9f50-b5081cb719ef`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:06:37,946::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:06:37,946::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:06:37,946::task::993::Storage.TaskManager.Task::(_decref) Task=`faec10e8-5c66-4905-9f50-b5081cb719ef`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:06:37,958::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:06:39,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:39,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913475 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:41,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:06:41,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:06:41,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:06:41,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000828929 s, 448 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:41,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108766 s, 9.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:06:41,621::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680031 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:43,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:43,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913524 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:06:44,627::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:06:44,628::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:06:45,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:45,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0969386 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:06:45,977::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:06:47,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:48,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913206 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:06:49,079::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:06:50,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:50,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912855 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:06:51,624::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:06:51,636::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:06:51,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656993 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:06:51,650::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685402 s, 763 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:52,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:52,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913402 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:06:53,103::task::595::Storage.TaskManager.Task::(_updateState) Task=`d384a9b8-7cf2-49dd-b379-90c902fc92c6`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:06:53,103::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:06:53,103::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656993', 'lastCheck': '1.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000685402', 'lastCheck': '1.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:06:53,104::task::1191::Storage.TaskManager.Task::(prepare) Task=`d384a9b8-7cf2-49dd-b379-90c902fc92c6`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656993', 'lastCheck': '1.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000685402', 'lastCheck': '1.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:06:53,104::task::595::Storage.TaskManager.Task::(_updateState) Task=`d384a9b8-7cf2-49dd-b379-90c902fc92c6`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:06:53,104::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:06:53,104::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:06:53,104::task::993::Storage.TaskManager.Task::(_decref) Task=`d384a9b8-7cf2-49dd-b379-90c902fc92c6`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:06:53,120::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:06:54,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:06:54,639::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:06:54,640::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:06:54,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913258 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:56,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:56,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914999 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:06:58,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:06:59,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912375 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:07:00,981::task::595::Storage.TaskManager.Task::(_updateState) Task=`42c73bb7-d876-4924-8996-3a1a05f309f2`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:07:00,981::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:07:00,984::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:07:00,984::task::1191::Storage.TaskManager.Task::(prepare) Task=`42c73bb7-d876-4924-8996-3a1a05f309f2`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:07:00,984::task::595::Storage.TaskManager.Task::(_updateState) Task=`42c73bb7-d876-4924-8996-3a1a05f309f2`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:07:00,984::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:07:00,984::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:07:00,984::task::993::Storage.TaskManager.Task::(_decref) Task=`42c73bb7-d876-4924-8996-3a1a05f309f2`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:07:01,012::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:07:01,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:01,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914565 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:07:01,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:07:01,667::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:07:01,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000925709 s, 401 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:07:01,680::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000611336 s, 856 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:03,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:03,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552662 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:07:04,649::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:07:04,651::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:07:05,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:05,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924245 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:07,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:07,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917827 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:07:08,212::task::595::Storage.TaskManager.Task::(_updateState) Task=`28ea1487-cc77-4c42-9804-a184fd5e993c`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:07:08,212::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:07:08,212::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000925709', 'lastCheck': '6.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000611336', 'lastCheck': '6.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:07:08,213::task::1191::Storage.TaskManager.Task::(prepare) Task=`28ea1487-cc77-4c42-9804-a184fd5e993c`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000925709', 'lastCheck': '6.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000611336', 'lastCheck': '6.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:07:08,213::task::595::Storage.TaskManager.Task::(_updateState) Task=`28ea1487-cc77-4c42-9804-a184fd5e993c`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:07:08,213::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:07:08,213::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:07:08,213::task::993::Storage.TaskManager.Task::(_decref) Task=`28ea1487-cc77-4c42-9804-a184fd5e993c`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:07:08,225::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:07:10,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:10,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915461 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:07:11,681::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:07:11,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000801042 s, 463 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:07:11,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:07:11,707::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000773259 s, 676 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:12,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:12,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914173 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:14,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:14,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915525 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:07:14,658::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:07:14,659::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:07:16,027::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:07:16,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:16,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0993359 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:18,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:07:19,110::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:07:19,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54783 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:21,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:21,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912422 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:07:21,708::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:07:21,720::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:07:21,734::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684022 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:07:21,736::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696176 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:23,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:07:23,329::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a291476-ffae-420f-adf1-71cf41c1500d`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:07:23,330::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:07:23,331::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000684022', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000696176', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:07:23,332::task::1191::Storage.TaskManager.Task::(prepare) Task=`4a291476-ffae-420f-adf1-71cf41c1500d`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000684022', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000696176', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:07:23,332::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a291476-ffae-420f-adf1-71cf41c1500d`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:07:23,333::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:07:23,334::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:07:23,334::task::993::Storage.TaskManager.Task::(_decref) Task=`4a291476-ffae-420f-adf1-71cf41c1500d`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:07:23,362::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:07:23,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.109847 s, 9.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:07:24,670::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:07:24,672::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:07:25,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:25,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913532 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:27,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:27,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132204 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:29,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:30,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544178 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:07:31,041::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 15:07:31,740::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:07:31,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:07:31,759::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648795 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:07:31,772::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:07:31,773::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:07:31,776::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:07:31,777::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:07:31,777::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:07:31,778::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:07:31,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:07:31,790::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645718 s, 810 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:32,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:32,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913071 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:34,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:34,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914284 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:07:34,683::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:07:34,684::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:07:36,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:36,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915012 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:07:38,452::task::595::Storage.TaskManager.Task::(_updateState) Task=`716e1bcf-3a88-482f-9b80-e1cffe67fe76`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:07:38,452::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:07:38,453::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648795', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000645718', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:07:38,453::task::1191::Storage.TaskManager.Task::(prepare) Task=`716e1bcf-3a88-482f-9b80-e1cffe67fe76`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648795', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000645718', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:07:38,453::task::595::Storage.TaskManager.Task::(_updateState) Task=`716e1bcf-3a88-482f-9b80-e1cffe67fe76`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:07:38,453::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:07:38,453::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:07:38,454::task::993::Storage.TaskManager.Task::(_decref) Task=`716e1bcf-3a88-482f-9b80-e1cffe67fe76`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:07:38,467::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:07:38,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:38,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914899 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:40,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:40,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915483 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:07:41,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:07:41,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656181 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:07:41,794::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:07:41,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672295 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:42,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:42,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913557 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:07:44,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:07:44,696::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:07:44,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:45,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914062 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:07:46,056::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:07:47,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:47,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913234 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:07:49,143::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:07:49,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:49,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914318 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:51,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:07:51,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:07:51,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:07:51,905::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655547 s, 566 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:07:51,906::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691604 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:51,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.595302 s, 1.7 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:07:53,561::task::595::Storage.TaskManager.Task::(_updateState) Task=`23376a5e-a239-481c-bba3-d0de159610e6`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:07:53,562::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:07:53,562::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000655547', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000691604', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:07:53,562::task::1191::Storage.TaskManager.Task::(prepare) Task=`23376a5e-a239-481c-bba3-d0de159610e6`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000655547', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000691604', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:07:53,562::task::595::Storage.TaskManager.Task::(_updateState) Task=`23376a5e-a239-481c-bba3-d0de159610e6`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:07:53,562::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:07:53,563::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:07:53,563::task::993::Storage.TaskManager.Task::(_decref) Task=`23376a5e-a239-481c-bba3-d0de159610e6`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:07:53,574::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:07:53,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:54,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912479 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:07:54,714::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:07:54,717::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:07:56,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:56,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915985 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:07:58,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:07:58,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914282 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:00,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:00,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913959 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:08:01,059::task::595::Storage.TaskManager.Task::(_updateState) Task=`807ab2b6-b5b6-43ac-9850-c2bc8007ba99`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:08:01,060::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:08:01,063::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:08:01,063::task::1191::Storage.TaskManager.Task::(prepare) Task=`807ab2b6-b5b6-43ac-9850-c2bc8007ba99`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:08:01,063::task::595::Storage.TaskManager.Task::(_updateState) Task=`807ab2b6-b5b6-43ac-9850-c2bc8007ba99`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:08:01,063::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:08:01,063::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:08:01,063::task::993::Storage.TaskManager.Task::(_decref) Task=`807ab2b6-b5b6-43ac-9850-c2bc8007ba99`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:08:01,092::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:08:01,913::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 15:08:01,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:08:01,940::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0007402 s, 707 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:08:01,950::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:08:01,952::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:08:01,955::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:08:01,956::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:08:01,957::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:08:01,957::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:08:01,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:08:01,969::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670271 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:02,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:02,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913989 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:04,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:04,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914093 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:08:04,716::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:08:04,717::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:06,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:06,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915056 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:08:08,662::task::595::Storage.TaskManager.Task::(_updateState) Task=`f47e0237-dc80-4fe6-bef2-64e55e0ae9a7`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:08:08,663::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:08:08,663::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000670271', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0007402', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:08:08,663::task::1191::Storage.TaskManager.Task::(prepare) Task=`f47e0237-dc80-4fe6-bef2-64e55e0ae9a7`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000670271', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0007402', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:08:08,663::task::595::Storage.TaskManager.Task::(_updateState) Task=`f47e0237-dc80-4fe6-bef2-64e55e0ae9a7`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:08:08,663::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:08:08,664::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:08:08,664::task::993::Storage.TaskManager.Task::(_decref) Task=`f47e0237-dc80-4fe6-bef2-64e55e0ae9a7`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:08:08,675::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:08,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:08,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915337 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:10,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:10,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915021 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:08:11,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:08:11,959::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679592 s, 770 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:08:11,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:08:11,984::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649886 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:12,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:13,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913925 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:08:14,728::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:08:14,729::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:15,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:15,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913817 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:08:16,110::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:08:17,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:17,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914235 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:08:19,176::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:08:19,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:19,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549789 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:21,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:08:22,487::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:08:22,500::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635597 s, 584 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:08:22,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:08:22,520::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713947 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:22,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.580509 s, 1.8 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:08:23,764::task::595::Storage.TaskManager.Task::(_updateState) Task=`2a216607-2b1c-4624-bba8-2bcf97b81cdd`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:08:23,764::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:08:23,765::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000635597', 'lastCheck': '1.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000713947', 'lastCheck': '1.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:08:23,765::task::1191::Storage.TaskManager.Task::(prepare) Task=`2a216607-2b1c-4624-bba8-2bcf97b81cdd`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000635597', 'lastCheck': '1.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000713947', 'lastCheck': '1.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:08:23,765::task::595::Storage.TaskManager.Task::(_updateState) Task=`2a216607-2b1c-4624-bba8-2bcf97b81cdd`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:08:23,765::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:08:23,765::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:08:23,765::task::993::Storage.TaskManager.Task::(_decref) Task=`2a216607-2b1c-4624-bba8-2bcf97b81cdd`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:08:23,777::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:24,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:24,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913685 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:08:24,741::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:08:24,742::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:26,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:27,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558246 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:29,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:29,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543318 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:08:31,121::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:08:31,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:31,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916656 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:08:32,508::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:08:32,520::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679774 s, 546 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:08:32,531::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:08:32,543::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000625366 s, 836 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:33,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:34,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543651 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:08:34,750::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:08:34,752::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:36,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:36,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091478 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:38,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:08:38,934::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f98a231-12ab-45c8-8b18-d84c6b7fb17c`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:08:38,934::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:08:38,934::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679774', 'lastCheck': '6.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000625366', 'lastCheck': '6.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:08:38,935::task::1191::Storage.TaskManager.Task::(prepare) Task=`5f98a231-12ab-45c8-8b18-d84c6b7fb17c`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679774', 'lastCheck': '6.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000625366', 'lastCheck': '6.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:08:38,935::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f98a231-12ab-45c8-8b18-d84c6b7fb17c`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:08:38,935::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:08:38,935::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:08:38,935::task::993::Storage.TaskManager.Task::(_decref) Task=`5f98a231-12ab-45c8-8b18-d84c6b7fb17c`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:08:38,947::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:39,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091235 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:41,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:41,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550177 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:08:42,524::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:08:42,536::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645976 s, 574 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:08:42,547::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:08:42,559::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645558 s, 810 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:43,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:43,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091556 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:08:44,758::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:08:44,759::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:45,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:45,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101793 s, 10.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:08:46,138::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:08:48,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:48,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913338 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:08:49,210::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:08:50,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:50,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915185 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:08:52,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:52,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915769 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:08:52,539::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:08:52,551::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065278 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:08:52,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:08:52,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657691 s, 795 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:08:54,043::task::595::Storage.TaskManager.Task::(_updateState) Task=`88b5bfb4-3a50-43ca-bca9-655581c5c731`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:08:54,044::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:08:54,044::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00065278', 'lastCheck': '1.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000657691', 'lastCheck': '1.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:08:54,044::task::1191::Storage.TaskManager.Task::(prepare) Task=`88b5bfb4-3a50-43ca-bca9-655581c5c731`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00065278', 'lastCheck': '1.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000657691', 'lastCheck': '1.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:08:54,044::task::595::Storage.TaskManager.Task::(_updateState) Task=`88b5bfb4-3a50-43ca-bca9-655581c5c731`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:08:54,044::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:08:54,044::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:08:54,045::task::993::Storage.TaskManager.Task::(_decref) Task=`88b5bfb4-3a50-43ca-bca9-655581c5c731`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:08:54,056::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:54,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:54,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920641 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:08:54,773::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:08:54,774::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:08:56,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:08:58,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37273 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:00,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:00,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921844 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:09:01,141::task::595::Storage.TaskManager.Task::(_updateState) Task=`9acbf2e3-b463-42dc-a3f3-93920638329f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:09:01,141::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:09:01,144::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:09:01,145::task::1191::Storage.TaskManager.Task::(prepare) Task=`9acbf2e3-b463-42dc-a3f3-93920638329f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:09:01,145::task::595::Storage.TaskManager.Task::(_updateState) Task=`9acbf2e3-b463-42dc-a3f3-93920638329f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:09:01,145::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:09:01,145::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:09:01,145::task::993::Storage.TaskManager.Task::(_decref) Task=`9acbf2e3-b463-42dc-a3f3-93920638329f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:09:01,173::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:09:02,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:02,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915375 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:09:02,554::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:09:02,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657522 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:09:02,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:09:02,589::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700413 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:04,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:04,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915568 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:09:04,785::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:09:04,786::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:09:06,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:06,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915535 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:08,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:08,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914177 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:09:09,146::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f140caa-8daa-4c75-a7dd-cfc3bf0f80d6`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:09:09,146::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:09:09,146::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000657522', 'lastCheck': '6.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000700413', 'lastCheck': '6.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:09:09,147::task::1191::Storage.TaskManager.Task::(prepare) Task=`4f140caa-8daa-4c75-a7dd-cfc3bf0f80d6`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000657522', 'lastCheck': '6.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000700413', 'lastCheck': '6.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:09:09,147::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f140caa-8daa-4c75-a7dd-cfc3bf0f80d6`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:09:09,147::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:09:09,147::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:09:09,147::task::993::Storage.TaskManager.Task::(_decref) Task=`4f140caa-8daa-4c75-a7dd-cfc3bf0f80d6`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:09:09,159::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:09:10,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:10,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924255 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:09:12,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:09:12,581::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00110724 s, 335 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:09:12,592::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:09:12,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666795 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:12,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:12,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914827 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:09:14,796::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:09:14,797::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:09:14,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:14,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914634 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:09:16,188::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:09:16,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:17,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915203 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:19,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:19,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911701 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:09:19,250::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:09:21,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:21,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091549 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:09:22,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:09:22,596::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006829 s, 543 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:09:22,607::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:09:22,619::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065764 s, 795 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:23,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:23,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914965 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:09:24,241::task::595::Storage.TaskManager.Task::(_updateState) Task=`0233e7c9-87c1-4b14-a613-9cdb854b9fcb`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:09:24,241::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:09:24,242::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0006829', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00065764', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:09:24,242::task::1191::Storage.TaskManager.Task::(prepare) Task=`0233e7c9-87c1-4b14-a613-9cdb854b9fcb`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0006829', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00065764', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:09:24,242::task::595::Storage.TaskManager.Task::(_updateState) Task=`0233e7c9-87c1-4b14-a613-9cdb854b9fcb`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:09:24,242::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:09:24,242::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:09:24,243::task::993::Storage.TaskManager.Task::(_decref) Task=`0233e7c9-87c1-4b14-a613-9cdb854b9fcb`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:09:24,254::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:09:24,805::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:09:24,806::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:09:25,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:25,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0971859 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:27,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:28,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555604 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:30,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:30,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549352 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:09:31,200::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:09:32,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:09:32,612::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652891 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:09:32,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:09:32,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649893 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:32,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:32,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915735 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:09:34,812::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:09:34,813::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:09:34,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:34,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913625 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:36,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:37,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915091 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:09:39,340::task::595::Storage.TaskManager.Task::(_updateState) Task=`ba1e59ab-017a-4516-89ff-bf76c56e36e2`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:09:39,340::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:09:39,341::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000652891', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000649893', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:09:39,341::task::1191::Storage.TaskManager.Task::(prepare) Task=`ba1e59ab-017a-4516-89ff-bf76c56e36e2`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000652891', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000649893', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:09:39,341::task::595::Storage.TaskManager.Task::(_updateState) Task=`ba1e59ab-017a-4516-89ff-bf76c56e36e2`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:09:39,341::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:09:39,341::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:09:39,341::task::993::Storage.TaskManager.Task::(_decref) Task=`ba1e59ab-017a-4516-89ff-bf76c56e36e2`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:09:39,354::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:09:39,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:39,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915868 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:41,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:41,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913655 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:09:42,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:09:42,627::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647384 s, 573 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:09:42,637::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:09:42,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000606439 s, 862 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:44,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:44,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091719 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:09:44,825::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:09:44,826::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:09:46,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:09:46,217::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:09:46,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50743 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:48,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:09:49,283::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:09:50,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.3655 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:52,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:52,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912109 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:09:52,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:09:52,642::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648223 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:09:52,652::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:09:52,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065597 s, 797 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:54,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:54,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091396 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:09:54,439::task::595::Storage.TaskManager.Task::(_updateState) Task=`9dcfb584-0342-4f97-aa32-4e75aa8f4737`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:09:54,440::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:09:54,440::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648223', 'lastCheck': '1.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00065597', 'lastCheck': '1.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:09:54,440::task::1191::Storage.TaskManager.Task::(prepare) Task=`9dcfb584-0342-4f97-aa32-4e75aa8f4737`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648223', 'lastCheck': '1.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00065597', 'lastCheck': '1.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:09:54,440::task::595::Storage.TaskManager.Task::(_updateState) Task=`9dcfb584-0342-4f97-aa32-4e75aa8f4737`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:09:54,440::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:09:54,441::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:09:54,441::task::993::Storage.TaskManager.Task::(_decref) Task=`9dcfb584-0342-4f97-aa32-4e75aa8f4737`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:09:54,452::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:09:54,836::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:09:54,837::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:09:56,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:56,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913066 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:09:58,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:09:58,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910551 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:00,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:10:01,220::task::595::Storage.TaskManager.Task::(_updateState) Task=`55e3c110-3607-4c0c-87b7-11c60084900f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:10:01,220::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:10:01,327::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:10:01,328::task::1191::Storage.TaskManager.Task::(prepare) Task=`55e3c110-3607-4c0c-87b7-11c60084900f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:10:01,329::task::595::Storage.TaskManager.Task::(_updateState) Task=`55e3c110-3607-4c0c-87b7-11c60084900f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:10:01,329::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:10:01,330::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:10:01,331::task::993::Storage.TaskManager.Task::(_decref) Task=`55e3c110-3607-4c0c-87b7-11c60084900f`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 15:10:01,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5454 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:10:01,367::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:10:02,645::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:10:02,657::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669391 s, 554 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:10:02,668::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:10:02,679::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666909 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:03,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:03,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914071 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:10:04,849::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:10:04,850::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:10:05,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:05,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913417 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:07,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:07,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917453 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:10:09,526::task::595::Storage.TaskManager.Task::(_updateState) Task=`53533afb-3b7d-461b-9d61-00cacf8a5720`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:10:09,526::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:10:09,526::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000669391', 'lastCheck': '6.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000666909', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:10:09,526::task::1191::Storage.TaskManager.Task::(prepare) Task=`53533afb-3b7d-461b-9d61-00cacf8a5720`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000669391', 'lastCheck': '6.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000666909', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:10:09,527::task::595::Storage.TaskManager.Task::(_updateState) Task=`53533afb-3b7d-461b-9d61-00cacf8a5720`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:10:09,527::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:10:09,527::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:10:09,527::task::993::Storage.TaskManager.Task::(_decref) Task=`53533afb-3b7d-461b-9d61-00cacf8a5720`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:10:09,539::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:10:09,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:09,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916651 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:11,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:12,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543662 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:10:12,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:10:12,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000600632 s, 618 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:10:12,683::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:10:12,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636602 s, 822 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:14,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:14,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912401 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:10:14,861::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:10:14,862::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:10:16,383::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:10:16,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:16,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909299 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:18,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:18,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091383 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:10:19,315::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:10:20,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:21,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539072 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:10:22,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:10:22,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655923 s, 566 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:10:22,698::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:10:22,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00073491 s, 712 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:23,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:23,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545573 s, 1.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:10:24,621::task::595::Storage.TaskManager.Task::(_updateState) Task=`3990d4f5-fcc5-444b-8ef8-4a4d388b81b7`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:10:24,621::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:10:24,621::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000655923', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00073491', 'lastCheck': '1.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:10:24,622::task::1191::Storage.TaskManager.Task::(prepare) Task=`3990d4f5-fcc5-444b-8ef8-4a4d388b81b7`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000655923', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00073491', 'lastCheck': '1.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:10:24,622::task::595::Storage.TaskManager.Task::(_updateState) Task=`3990d4f5-fcc5-444b-8ef8-4a4d388b81b7`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:10:24,622::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:10:24,622::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:10:24,622::task::993::Storage.TaskManager.Task::(_decref) Task=`3990d4f5-fcc5-444b-8ef8-4a4d388b81b7`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:10:24,634::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:10:24,872::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:10:24,873::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:10:25,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:25,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913493 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:27,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:28,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544177 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:30,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:31,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541992 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:10:31,397::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:10:32,690::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:10:32,702::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664302 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:10:32,714::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:10:32,726::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686951 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:33,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:33,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915391 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:10:34,882::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:10:34,883::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:10:35,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:35,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132463 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:37,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:37,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545365 s, 1.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:10:39,730::task::595::Storage.TaskManager.Task::(_updateState) Task=`85c5309d-5c92-4097-96c0-2999be341ed9`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:10:39,730::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:10:39,731::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000664302', 'lastCheck': '7.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000686951', 'lastCheck': '7.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:10:39,731::task::1191::Storage.TaskManager.Task::(prepare) Task=`85c5309d-5c92-4097-96c0-2999be341ed9`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000664302', 'lastCheck': '7.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000686951', 'lastCheck': '7.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:10:39,731::task::595::Storage.TaskManager.Task::(_updateState) Task=`85c5309d-5c92-4097-96c0-2999be341ed9`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:10:39,731::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:10:39,731::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:10:39,732::task::993::Storage.TaskManager.Task::(_decref) Task=`85c5309d-5c92-4097-96c0-2999be341ed9`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:10:39,743::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:10:39,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:40,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914586 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:42,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:42,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558009 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:10:42,706::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:10:42,718::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063234 s, 587 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:10:42,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:10:42,742::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657619 s, 795 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:44,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:44,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091367 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:10:44,893::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:10:44,895::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:10:46,412::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:10:46,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:46,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914316 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:48,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:49,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914436 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:10:49,346::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:10:51,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:51,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091376 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:10:52,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:10:52,734::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648455 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:10:52,745::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:10:52,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679351 s, 770 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:53,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:53,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09139 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:10:54,836::task::595::Storage.TaskManager.Task::(_updateState) Task=`0fb8332c-783a-4307-b80e-590a45478654`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:10:54,836::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:10:54,836::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648455', 'lastCheck': '2.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679351', 'lastCheck': '2.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:10:54,837::task::1191::Storage.TaskManager.Task::(prepare) Task=`0fb8332c-783a-4307-b80e-590a45478654`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648455', 'lastCheck': '2.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679351', 'lastCheck': '2.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:10:54,837::task::595::Storage.TaskManager.Task::(_updateState) Task=`0fb8332c-783a-4307-b80e-590a45478654`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:10:54,837::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:10:54,837::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:10:54,837::task::993::Storage.TaskManager.Task::(_decref) Task=`0fb8332c-783a-4307-b80e-590a45478654`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:10:54,859::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:10:54,902::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:10:54,903::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:10:55,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:55,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912516 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:57,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:57,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913682 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:10:59,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:10:59,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912264 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:11:01,416::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5723b3b-a159-4270-9e51-9a069755e52f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:11:01,417::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:11:01,419::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:11:01,419::task::1191::Storage.TaskManager.Task::(prepare) Task=`f5723b3b-a159-4270-9e51-9a069755e52f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:11:01,420::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5723b3b-a159-4270-9e51-9a069755e52f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:11:01,420::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:11:01,420::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:11:01,420::task::993::Storage.TaskManager.Task::(_decref) Task=`f5723b3b-a159-4270-9e51-9a069755e52f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:11:01,456::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:11:01,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:02,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511709 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:11:02,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:11:02,749::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650584 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:11:02,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:11:02,772::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679463 s, 770 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:04,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:11:04,913::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:11:04,914::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:05,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36384 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:07,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:08,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37639 s, 744 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:11:09,941::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b15a2a6-8eb6-4a03-9862-111fd104649b`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:11:09,941::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:11:09,941::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650584', 'lastCheck': '7.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679463', 'lastCheck': '7.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:11:09,941::task::1191::Storage.TaskManager.Task::(prepare) Task=`3b15a2a6-8eb6-4a03-9862-111fd104649b`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650584', 'lastCheck': '7.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679463', 'lastCheck': '7.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:11:09,942::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b15a2a6-8eb6-4a03-9862-111fd104649b`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:11:09,942::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:11:09,942::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:11:09,942::task::993::Storage.TaskManager.Task::(_decref) Task=`3b15a2a6-8eb6-4a03-9862-111fd104649b`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:11:09,954::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:10,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:11,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927581 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:11:12,753::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:11:12,765::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653203 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:11:12,776::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:11:12,787::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670599 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:13,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:13,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558273 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:11:14,922::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:11:14,923::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:15,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:15,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968784 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:11:16,470::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:11:17,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:17,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913194 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:11:19,380::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:11:19,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:20,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914816 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:22,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:22,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913345 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:11:22,769::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:11:22,780::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000698187 s, 531 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:11:22,791::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:11:22,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00061672 s, 848 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:24,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:24,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547454 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:11:24,932::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:11:24,933::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 15:11:25,035::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3740ff0-797b-4472-b031-c29744de9d49`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:11:25,035::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:11:25,035::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000698187', 'lastCheck': '2.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00061672', 'lastCheck': '2.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:11:25,035::task::1191::Storage.TaskManager.Task::(prepare) Task=`a3740ff0-797b-4472-b031-c29744de9d49`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000698187', 'lastCheck': '2.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00061672', 'lastCheck': '2.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:11:25,035::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3740ff0-797b-4472-b031-c29744de9d49`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:11:25,036::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:11:25,036::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:11:25,036::task::993::Storage.TaskManager.Task::(_decref) Task=`a3740ff0-797b-4472-b031-c29744de9d49`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:11:25,047::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:26,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:26,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913793 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:28,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:29,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561033 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:11:31,483::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:11:31,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:31,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914469 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:11:32,783::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:11:32,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674783 s, 550 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:11:32,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:11:32,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691394 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:33,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:33,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914031 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:11:34,943::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:11:34,944::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:35,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:35,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914558 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:37,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:37,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914584 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:39,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:11:40,129::task::595::Storage.TaskManager.Task::(_updateState) Task=`ad8e16cd-8409-4760-a087-862a66093ca3`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:11:40,130::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:11:40,130::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000674783', 'lastCheck': '7.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000691394', 'lastCheck': '7.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:11:40,130::task::1191::Storage.TaskManager.Task::(prepare) Task=`ad8e16cd-8409-4760-a087-862a66093ca3`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000674783', 'lastCheck': '7.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000691394', 'lastCheck': '7.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:11:40,131::task::595::Storage.TaskManager.Task::(_updateState) Task=`ad8e16cd-8409-4760-a087-862a66093ca3`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:11:40,131::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:11:40,131::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:11:40,131::task::993::Storage.TaskManager.Task::(_decref) Task=`ad8e16cd-8409-4760-a087-862a66093ca3`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:11:40,144::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:40,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.553244 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:42,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:11:42,822::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:11:42,835::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:11:43,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000735935 s, 711 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:11:43,367::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000940678 s, 394 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:43,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555728 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:11:44,952::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:11:44,953::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:45,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:46,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912156 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:11:46,499::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:11:48,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:48,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550539 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:11:49,412::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:11:50,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:50,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911436 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:52,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:52,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091365 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:11:53,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:11:53,387::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:11:53,402::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000633435 s, 586 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:11:53,404::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645937 s, 810 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:11:54,963::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:54,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:11:54,976::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:55,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913989 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:11:55,234::task::595::Storage.TaskManager.Task::(_updateState) Task=`947d6495-71e4-4341-9f35-8bdd8d1ed626`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:11:55,234::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:11:55,234::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000633435', 'lastCheck': '1.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000645937', 'lastCheck': '1.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:11:55,234::task::1191::Storage.TaskManager.Task::(prepare) Task=`947d6495-71e4-4341-9f35-8bdd8d1ed626`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000633435', 'lastCheck': '1.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000645937', 'lastCheck': '1.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:11:55,235::task::595::Storage.TaskManager.Task::(_updateState) Task=`947d6495-71e4-4341-9f35-8bdd8d1ed626`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:11:55,235::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:11:55,235::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:11:55,235::task::993::Storage.TaskManager.Task::(_decref) Task=`947d6495-71e4-4341-9f35-8bdd8d1ed626`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:11:55,246::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:11:57,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:57,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912514 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:11:59,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:11:59,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914577 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:01,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:01,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914507 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:12:01,502::task::595::Storage.TaskManager.Task::(_updateState) Task=`000ffcaf-403c-4951-ab09-46038dbe4d98`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:12:01,503::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:12:01,506::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:12:01,506::task::1191::Storage.TaskManager.Task::(prepare) Task=`000ffcaf-403c-4951-ab09-46038dbe4d98`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:12:01,506::task::595::Storage.TaskManager.Task::(_updateState) Task=`000ffcaf-403c-4951-ab09-46038dbe4d98`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:12:01,506::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:12:01,506::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:12:01,507::task::993::Storage.TaskManager.Task::(_decref) Task=`000ffcaf-403c-4951-ab09-46038dbe4d98`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:12:01,535::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:12:03,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:12:03,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:12:03,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00411786 s, 90.1 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:03,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:12:03,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00118355 s, 442 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:03,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912377 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:12:04,974::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:12:04,975::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:12:05,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:06,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557015 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:08,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:08,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913583 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:10,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:12:10,335::task::595::Storage.TaskManager.Task::(_updateState) Task=`e2b94112-ffda-4091-8086-eedc87a0da24`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:12:10,336::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:12:10,336::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00411786', 'lastCheck': '6.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00118355', 'lastCheck': '6.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:12:10,337::task::1191::Storage.TaskManager.Task::(prepare) Task=`e2b94112-ffda-4091-8086-eedc87a0da24`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00411786', 'lastCheck': '6.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00118355', 'lastCheck': '6.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:12:10,337::task::595::Storage.TaskManager.Task::(_updateState) Task=`e2b94112-ffda-4091-8086-eedc87a0da24`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:12:10,337::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:12:10,337::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:12:10,337::task::993::Storage.TaskManager.Task::(_decref) Task=`e2b94112-ffda-4091-8086-eedc87a0da24`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:12:10,355::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:12:10,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913348 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:12,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:12:13,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:12:13,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:12:13,793::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626503 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:12:13,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635739 s, 823 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:13,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.42156 s, 720 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:12:14,979::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:12:14,980::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:12:15,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:15,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108063 s, 9.5 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:12:16,550::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:12:17,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:18,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091335 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:12:19,445::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:12:20,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:20,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912541 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:22,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:22,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913068 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:12:23,804::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:12:23,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:12:23,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00075067 s, 494 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:12:23,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650828 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:24,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:24,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551828 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:12:24,993::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:12:25,011::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 15:12:25,439::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d3aa9e9-cad0-49a4-a10b-8bf9807e0d81`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:12:25,439::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:12:25,440::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00075067', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650828', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:12:25,440::task::1191::Storage.TaskManager.Task::(prepare) Task=`5d3aa9e9-cad0-49a4-a10b-8bf9807e0d81`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00075067', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650828', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:12:25,440::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d3aa9e9-cad0-49a4-a10b-8bf9807e0d81`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:12:25,440::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:12:25,440::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:12:25,440::task::993::Storage.TaskManager.Task::(_decref) Task=`5d3aa9e9-cad0-49a4-a10b-8bf9807e0d81`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:12:25,451::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:12:27,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:27,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910826 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:29,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:29,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091343 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:31,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:31,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914356 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:12:31,563::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:12:33,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:33,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918983 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:12:33,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:12:33,842::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:12:33,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660052 s, 562 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:12:33,864::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:12:33,865::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:12:33,868::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:12:33,869::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:12:33,870::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:12:33,870::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:12:33,871::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:12:33,882::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648912 s, 806 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:12:35,003::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:12:35,004::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:12:35,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:35,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922965 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:37,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:37,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914255 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:39,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:39,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091412 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:12:40,524::task::595::Storage.TaskManager.Task::(_updateState) Task=`062c94aa-a25a-42a6-b84b-70f518bfa40d`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:12:40,525::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:12:40,525::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000660052', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648912', 'lastCheck': '6.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:12:40,525::task::1191::Storage.TaskManager.Task::(prepare) Task=`062c94aa-a25a-42a6-b84b-70f518bfa40d`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000660052', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000648912', 'lastCheck': '6.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:12:40,525::task::595::Storage.TaskManager.Task::(_updateState) Task=`062c94aa-a25a-42a6-b84b-70f518bfa40d`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:12:40,525::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:12:40,525::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:12:40,526::task::993::Storage.TaskManager.Task::(_decref) Task=`062c94aa-a25a-42a6-b84b-70f518bfa40d`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:12:40,537::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:12:41,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:42,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913987 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:12:43,859::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:12:43,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643878 s, 576 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:12:43,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:12:43,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652975 s, 801 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:44,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:44,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915914 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:12:45,010::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:12:45,012::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:12:46,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:46,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0971198 s, 10.5 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:12:46,576::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:12:48,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:48,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913965 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:12:49,482::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:12:50,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:50,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912296 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:52,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:52,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921405 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:12:53,875::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:12:53,887::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652903 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:12:53,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:12:53,912::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684172 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:54,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:54,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913416 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:12:55,023::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:12:55,024::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 15:12:55,629::task::595::Storage.TaskManager.Task::(_updateState) Task=`39cf3876-b8e3-41e6-a79b-cc4697175288`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:12:55,630::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:12:55,630::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000652903', 'lastCheck': '1.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000684172', 'lastCheck': '1.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:12:55,630::task::1191::Storage.TaskManager.Task::(prepare) Task=`39cf3876-b8e3-41e6-a79b-cc4697175288`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000652903', 'lastCheck': '1.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000684172', 'lastCheck': '1.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:12:55,630::task::595::Storage.TaskManager.Task::(_updateState) Task=`39cf3876-b8e3-41e6-a79b-cc4697175288`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:12:55,631::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:12:55,631::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:12:55,631::task::993::Storage.TaskManager.Task::(_decref) Task=`39cf3876-b8e3-41e6-a79b-cc4697175288`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:12:55,643::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:12:56,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:57,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544458 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:12:59,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:12:59,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919562 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:01,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:13:01,579::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7b39164-8d89-41a2-a234-bf3fd0431f1f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:13:01,579::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:13:01,658::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:13:01,658::task::1191::Storage.TaskManager.Task::(prepare) Task=`c7b39164-8d89-41a2-a234-bf3fd0431f1f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:13:01,658::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7b39164-8d89-41a2-a234-bf3fd0431f1f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:13:01,659::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:13:01,659::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:13:01,659::task::993::Storage.TaskManager.Task::(_decref) Task=`c7b39164-8d89-41a2-a234-bf3fd0431f1f`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 15:13:01,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0947794 s, 10.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:13:01,691::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:13:03,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:13:03,889::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Dummy-49::DEBUG::2014-07-14 15:13:03,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.22954 s, 4.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:13:03,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:13:03,945::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675553 s, 774 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:13:03,969::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:13:03,971::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:13:03,975::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:13:03,978::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:13:03,979::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:13:03,980::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:13:03,984::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:13:03,996::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000754393 s, 492 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:13:05,035::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:13:05,036::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:13:06,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:06,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914151 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:08,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:08,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542752 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:10,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:13:10,735::task::595::Storage.TaskManager.Task::(_updateState) Task=`e44ed46c-b9da-4ace-bfaf-376254da5e00`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:13:10,736::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:13:10,736::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000754393', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000675553', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:13:10,736::task::1191::Storage.TaskManager.Task::(prepare) Task=`e44ed46c-b9da-4ace-bfaf-376254da5e00`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000754393', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000675553', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:13:10,736::task::595::Storage.TaskManager.Task::(_updateState) Task=`e44ed46c-b9da-4ace-bfaf-376254da5e00`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:13:10,737::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:13:10,737::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:13:10,737::task::993::Storage.TaskManager.Task::(_decref) Task=`e44ed46c-b9da-4ace-bfaf-376254da5e00`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:13:10,753::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:13:10,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915704 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:12,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:12,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915036 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:13:13,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:13:13,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695545 s, 752 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:13:14,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:13:14,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690771 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:14,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:15,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915943 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:13:15,049::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:13:15,051::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:13:16,704::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:13:17,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:17,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921206 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:19,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:19,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091328 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:13:19,515::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:13:21,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:21,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091632 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:23,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:23,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913218 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:13:23,973::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:13:23,985::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689319 s, 759 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:13:24,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:13:24,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669719 s, 554 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:13:25,059::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:13:25,060::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:13:25,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:25,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913889 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:13:25,834::task::595::Storage.TaskManager.Task::(_updateState) Task=`23c198ca-2a6c-478d-a89e-e9896c7940cd`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:13:25,835::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:13:25,835::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000669719', 'lastCheck': '1.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000689319', 'lastCheck': '1.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:13:25,835::task::1191::Storage.TaskManager.Task::(prepare) Task=`23c198ca-2a6c-478d-a89e-e9896c7940cd`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000669719', 'lastCheck': '1.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000689319', 'lastCheck': '1.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:13:25,835::task::595::Storage.TaskManager.Task::(_updateState) Task=`23c198ca-2a6c-478d-a89e-e9896c7940cd`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:13:25,836::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:13:25,836::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:13:25,836::task::993::Storage.TaskManager.Task::(_decref) Task=`23c198ca-2a6c-478d-a89e-e9896c7940cd`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:13:25,847::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:13:27,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:27,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912612 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:29,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:29,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913167 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:13:31,717::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:13:31,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:31,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913406 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:13:33,989::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:34,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:13:34,010::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666999 s, 784 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:13:34,101::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:34,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10337 s, 9.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:13:34,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623697 s, 595 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:13:35,069::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:13:35,070::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:13:36,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:36,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555672 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:38,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:38,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916962 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:40,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:40,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914858 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:13:40,944::task::595::Storage.TaskManager.Task::(_updateState) Task=`61dc66da-8893-43c5-a0d2-700a9ab960c7`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:13:40,944::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:13:40,945::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000623697', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000666999', 'lastCheck': '6.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:13:40,947::task::1191::Storage.TaskManager.Task::(prepare) Task=`61dc66da-8893-43c5-a0d2-700a9ab960c7`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000623697', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000666999', 'lastCheck': '6.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:13:40,948::task::595::Storage.TaskManager.Task::(_updateState) Task=`61dc66da-8893-43c5-a0d2-700a9ab960c7`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:13:40,948::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:13:40,949::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:13:40,949::task::993::Storage.TaskManager.Task::(_decref) Task=`61dc66da-8893-43c5-a0d2-700a9ab960c7`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:13:40,970::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:13:42,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:43,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915527 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:13:44,015::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:13:44,027::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672984 s, 777 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:13:44,125::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:13:44,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621258 s, 597 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:13:45,079::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:13:45,080::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:13:45,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:45,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559503 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:13:46,729::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:13:47,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:48,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542724 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:13:49,547::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:13:50,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:50,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913199 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:52,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:52,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091237 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:13:54,031::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:13:54,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000711072 s, 736 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:13:54,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:13:54,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654272 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:54,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:54,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912109 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:13:55,085::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:13:55,086::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 15:13:56,092::task::595::Storage.TaskManager.Task::(_updateState) Task=`23221eaf-ad92-4de5-b46a-8b2014b2c3c2`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:13:56,093::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:13:56,093::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000654272', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000711072', 'lastCheck': '2.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:13:56,093::task::1191::Storage.TaskManager.Task::(prepare) Task=`23221eaf-ad92-4de5-b46a-8b2014b2c3c2`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000654272', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000711072', 'lastCheck': '2.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:13:56,093::task::595::Storage.TaskManager.Task::(_updateState) Task=`23221eaf-ad92-4de5-b46a-8b2014b2c3c2`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:13:56,093::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:13:56,094::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:13:56,094::task::993::Storage.TaskManager.Task::(_decref) Task=`23221eaf-ad92-4de5-b46a-8b2014b2c3c2`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:13:56,112::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:13:56,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:57,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559557 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:13:59,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:13:59,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544966 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:14:01,732::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3ef58a6-111e-4ed4-a034-17c3fb2c68b5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:14:01,732::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:14:01,735::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:14:01,735::task::1191::Storage.TaskManager.Task::(prepare) Task=`e3ef58a6-111e-4ed4-a034-17c3fb2c68b5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:14:01,735::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3ef58a6-111e-4ed4-a034-17c3fb2c68b5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:14:01,736::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:14:01,736::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:14:01,736::task::993::Storage.TaskManager.Task::(_decref) Task=`e3ef58a6-111e-4ed4-a034-17c3fb2c68b5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:14:01,764::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:14:01,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:01,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914573 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:03,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:14:04,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:14:04,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:14:04,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00268823 s, 138 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:14:04,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00222192 s, 235 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:04,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541069 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:14:05,095::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:14:05,096::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:14:06,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:06,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926807 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:08,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:08,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912945 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:10,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:14:11,196::task::595::Storage.TaskManager.Task::(_updateState) Task=`4772ad0a-9a9f-4c81-8159-1e6fea8a45e5`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:14:11,196::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:14:11,197::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00268823', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00222192', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:14:11,197::task::1191::Storage.TaskManager.Task::(prepare) Task=`4772ad0a-9a9f-4c81-8159-1e6fea8a45e5`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00268823', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00222192', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:14:11,197::task::595::Storage.TaskManager.Task::(_updateState) Task=`4772ad0a-9a9f-4c81-8159-1e6fea8a45e5`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:14:11,197::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:14:11,197::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:14:11,197::task::993::Storage.TaskManager.Task::(_decref) Task=`4772ad0a-9a9f-4c81-8159-1e6fea8a45e5`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:14:11,209::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:14:11,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914433 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:13,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:14,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914621 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:14:14,520::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:14:14,532::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627469 s, 591 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:14:14,533::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:14:14,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648109 s, 807 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:14:15,108::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:14:15,109::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:14:16,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:16,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100124 s, 10.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:14:16,775::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:14:18,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:18,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916105 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:14:19,580::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:14:20,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:20,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914529 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:22,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:22,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913384 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:14:24,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:14:24,559::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:14:24,574::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625658 s, 593 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:14:24,575::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635177 s, 823 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:24,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:24,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913752 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:14:25,120::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:14:25,126::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 15:14:26,300::task::595::Storage.TaskManager.Task::(_updateState) Task=`31bcdd5d-d07f-4dc9-b788-42504d2403df`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:14:26,300::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:14:26,300::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000625658', 'lastCheck': '1.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000635177', 'lastCheck': '1.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:14:26,301::task::1191::Storage.TaskManager.Task::(prepare) Task=`31bcdd5d-d07f-4dc9-b788-42504d2403df`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000625658', 'lastCheck': '1.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000635177', 'lastCheck': '1.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:14:26,301::task::595::Storage.TaskManager.Task::(_updateState) Task=`31bcdd5d-d07f-4dc9-b788-42504d2403df`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:14:26,301::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:14:26,301::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:14:26,301::task::993::Storage.TaskManager.Task::(_decref) Task=`31bcdd5d-d07f-4dc9-b788-42504d2403df`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:14:26,313::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:14:26,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:26,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913167 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:28,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:28,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091305 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:30,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:31,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915216 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:14:31,787::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:14:33,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:33,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556766 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:14:34,580::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:14:34,591::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:14:34,603::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000776615 s, 478 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:14:34,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629477 s, 831 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:14:35,129::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:14:35,130::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:14:35,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:35,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914841 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:37,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:37,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913573 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:39,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:40,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543929 s, 1.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:14:41,422::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a2ecc28-32a1-4f4a-b021-cbfc144fab64`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:14:41,423::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:14:41,423::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000776615', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000629477', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:14:41,423::task::1191::Storage.TaskManager.Task::(prepare) Task=`0a2ecc28-32a1-4f4a-b021-cbfc144fab64`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000776615', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000629477', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:14:41,423::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a2ecc28-32a1-4f4a-b021-cbfc144fab64`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:14:41,424::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:14:41,424::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:14:41,424::task::993::Storage.TaskManager.Task::(_decref) Task=`0a2ecc28-32a1-4f4a-b021-cbfc144fab64`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:14:41,439::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:14:42,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:42,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091447 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:44,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:14:44,617::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:14:44,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:14:44,708::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657156 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:14:44,707::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000904352 s, 578 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:44,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100453 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:14:45,141::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:14:45,142::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:14:46,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:14:46,801::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:14:46,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09165 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:48,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:49,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519218 s, 2.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:14:49,621::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:14:51,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:52,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36358 s, 751 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:14:54,714::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:14:54,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:14:54,737::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625029 s, 594 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:14:54,739::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000708798 s, 738 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:54,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:54,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913135 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:14:55,153::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:14:55,154::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-11142::DEBUG::2014-07-14 15:14:56,521::task::595::Storage.TaskManager.Task::(_updateState) Task=`75835383-d525-4ad3-972c-5d03a4007331`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:14:56,522::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:14:56,522::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000625029', 'lastCheck': '1.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000708798', 'lastCheck': '1.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:14:56,523::task::1191::Storage.TaskManager.Task::(prepare) Task=`75835383-d525-4ad3-972c-5d03a4007331`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000625029', 'lastCheck': '1.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000708798', 'lastCheck': '1.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:14:56,523::task::595::Storage.TaskManager.Task::(_updateState) Task=`75835383-d525-4ad3-972c-5d03a4007331`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:14:56,523::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:14:56,523::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:14:56,523::task::993::Storage.TaskManager.Task::(_decref) Task=`75835383-d525-4ad3-972c-5d03a4007331`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:14:56,535::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:14:56,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:57,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911777 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:14:59,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:14:59,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914216 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:01,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:01,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913866 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:15:01,803::task::595::Storage.TaskManager.Task::(_updateState) Task=`b55f486f-2f56-49ee-bb63-c08351d2160b`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:15:01,804::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:15:01,807::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:15:01,807::task::1191::Storage.TaskManager.Task::(prepare) Task=`b55f486f-2f56-49ee-bb63-c08351d2160b`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:15:01,808::task::595::Storage.TaskManager.Task::(_updateState) Task=`b55f486f-2f56-49ee-bb63-c08351d2160b`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:15:01,808::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:15:01,808::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:15:01,808::task::993::Storage.TaskManager.Task::(_decref) Task=`b55f486f-2f56-49ee-bb63-c08351d2160b`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:15:01,836::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:15:03,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:03,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914766 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:15:04,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:15:04,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0013744 s, 270 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:15:04,762::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:15:04,775::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000715346 s, 731 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:15:05,162::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:15:05,163::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:15:05,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:05,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914104 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:07,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:07,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914014 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:09,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:09,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0941374 s, 10.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:15:11,620::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec597dcf-d631-4fad-800f-36ec45432ca0`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:15:11,620::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:15:11,620::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0013744', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000715346', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:15:11,621::task::1191::Storage.TaskManager.Task::(prepare) Task=`ec597dcf-d631-4fad-800f-36ec45432ca0`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0013744', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000715346', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:15:11,621::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec597dcf-d631-4fad-800f-36ec45432ca0`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:15:11,621::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:15:11,621::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:15:11,621::task::993::Storage.TaskManager.Task::(_decref) Task=`ec597dcf-d631-4fad-800f-36ec45432ca0`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:15:11,642::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:15:11,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:11,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915401 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:13,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:13,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915569 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:15:15,174::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:15:15,175::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 15:15:15,355::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:15:15,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:15:15,382::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000634482 s, 824 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:15:15,383::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622957 s, 596 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:15,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:16,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100975 s, 10.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:15:16,849::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:15:18,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:18,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091329 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:15:19,651::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:15:20,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:20,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549447 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:22,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:23,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913834 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:25,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:25,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914168 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:15:25,186::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:15:25,187::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:15:25,391::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:15:25,402::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:15:25,413::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00509675 s, 72.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:15:25,416::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00101676 s, 514 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:15:26,735::task::595::Storage.TaskManager.Task::(_updateState) Task=`8be9c2dd-6c15-4163-8446-e146dcc48d4f`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:15:26,735::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:15:26,735::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00509675', 'lastCheck': '1.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00101676', 'lastCheck': '1.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:15:26,735::task::1191::Storage.TaskManager.Task::(prepare) Task=`8be9c2dd-6c15-4163-8446-e146dcc48d4f`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00509675', 'lastCheck': '1.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00101676', 'lastCheck': '1.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:15:26,736::task::595::Storage.TaskManager.Task::(_updateState) Task=`8be9c2dd-6c15-4163-8446-e146dcc48d4f`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:15:26,736::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:15:26,736::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:15:26,736::task::993::Storage.TaskManager.Task::(_decref) Task=`8be9c2dd-6c15-4163-8446-e146dcc48d4f`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:15:26,748::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:15:27,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:27,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542632 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:29,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:31,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37457 s, 745 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:15:31,862::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:15:33,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:33,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916854 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:15:35,197::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:15:35,198::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:15:35,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:15:35,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:15:35,484::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:15:35,496::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00267453 s, 139 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:15:35,497::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657529 s, 795 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:35,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.244732 s, 4.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:37,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:37,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914335 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:39,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:39,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909562 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:41,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:15:41,851::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c0c973c-5a77-48b6-82e8-f2fd8624d9f7`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:15:41,851::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:15:41,851::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00267453', 'lastCheck': '6.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000657529', 'lastCheck': '6.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:15:41,852::task::1191::Storage.TaskManager.Task::(prepare) Task=`6c0c973c-5a77-48b6-82e8-f2fd8624d9f7`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00267453', 'lastCheck': '6.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000657529', 'lastCheck': '6.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:15:41,852::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c0c973c-5a77-48b6-82e8-f2fd8624d9f7`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:15:41,852::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:15:41,852::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:15:41,852::task::993::Storage.TaskManager.Task::(_decref) Task=`6c0c973c-5a77-48b6-82e8-f2fd8624d9f7`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 15:15:41,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09164 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::ERROR::2014-07-14 15:15:41,871::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:15:43,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:43,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913891 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:15:45,209::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:15:45,210::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:15:45,504::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:15:45,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:15:45,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000801086 s, 463 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:15:45,530::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000740227 s, 707 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:45,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:46,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091149 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:15:46,875::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:15:48,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:48,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914019 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:15:49,683::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:15:50,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:50,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913331 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:52,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:52,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913147 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:54,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:54,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913753 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:15:55,221::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:15:55,222::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:15:55,532::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:15:55,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:15:55,556::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629315 s, 590 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:15:55,559::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000731662 s, 715 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:15:56,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:56,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913821 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:15:56,967::task::595::Storage.TaskManager.Task::(_updateState) Task=`49ed3851-7c05-4bc5-a4dd-82e5f61662fb`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:15:56,968::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:15:56,968::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000629315', 'lastCheck': '1.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000731662', 'lastCheck': '1.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:15:56,968::task::1191::Storage.TaskManager.Task::(prepare) Task=`49ed3851-7c05-4bc5-a4dd-82e5f61662fb`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000629315', 'lastCheck': '1.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000731662', 'lastCheck': '1.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:15:56,968::task::595::Storage.TaskManager.Task::(_updateState) Task=`49ed3851-7c05-4bc5-a4dd-82e5f61662fb`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:15:56,968::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:15:56,969::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:15:56,969::task::993::Storage.TaskManager.Task::(_decref) Task=`49ed3851-7c05-4bc5-a4dd-82e5f61662fb`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:15:56,980::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:15:58,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:15:58,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913631 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:00,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:01,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915727 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:16:01,878::task::595::Storage.TaskManager.Task::(_updateState) Task=`03ae4d9f-e601-4318-b356-95c22377aa51`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:16:01,878::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:16:01,881::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:16:01,881::task::1191::Storage.TaskManager.Task::(prepare) Task=`03ae4d9f-e601-4318-b356-95c22377aa51`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:16:01,881::task::595::Storage.TaskManager.Task::(_updateState) Task=`03ae4d9f-e601-4318-b356-95c22377aa51`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:16:01,881::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:16:01,882::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:16:01,882::task::993::Storage.TaskManager.Task::(_decref) Task=`03ae4d9f-e601-4318-b356-95c22377aa51`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:16:01,909::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:16:03,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:03,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914646 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:05,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:16:05,233::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:16:05,235::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:16:05,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913851 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:16:05,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:16:05,574::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:16:05,590::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626615 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:16:05,592::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000608917 s, 859 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:07,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:07,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914279 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:09,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:09,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915211 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:11,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:11,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913969 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:16:12,063::task::595::Storage.TaskManager.Task::(_updateState) Task=`b9c754cb-5d8b-4500-bb93-a6b95c63234c`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:16:12,063::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:16:12,063::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000626615', 'lastCheck': '6.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000608917', 'lastCheck': '6.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:16:12,064::task::1191::Storage.TaskManager.Task::(prepare) Task=`b9c754cb-5d8b-4500-bb93-a6b95c63234c`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000626615', 'lastCheck': '6.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000608917', 'lastCheck': '6.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:16:12,064::task::595::Storage.TaskManager.Task::(_updateState) Task=`b9c754cb-5d8b-4500-bb93-a6b95c63234c`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:16:12,064::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:16:12,064::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:16:12,064::task::993::Storage.TaskManager.Task::(_decref) Task=`b9c754cb-5d8b-4500-bb93-a6b95c63234c`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:16:12,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:16:13,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:13,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913949 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:16:15,242::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:16:15,243::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:16:15,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:16:15,601::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:16:15,629::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686341 s, 541 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:16:15,632::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000738278 s, 708 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:15,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:15,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999792 s, 10.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:16:16,922::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:16:18,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:18,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914967 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:16:19,716::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:16:20,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:20,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914131 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:22,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:22,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915663 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:24,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:24,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913739 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:16:25,254::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:16:25,255::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:16:25,641::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:16:25,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000736411 s, 504 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:16:25,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:16:25,667::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651059 s, 803 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:26,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:26,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091518 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:16:27,157::task::595::Storage.TaskManager.Task::(_updateState) Task=`51af7346-2608-4f42-8a0e-6594b38b76c2`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:16:27,158::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:16:27,158::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000736411', 'lastCheck': '1.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000651059', 'lastCheck': '1.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:16:27,158::task::1191::Storage.TaskManager.Task::(prepare) Task=`51af7346-2608-4f42-8a0e-6594b38b76c2`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000736411', 'lastCheck': '1.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000651059', 'lastCheck': '1.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:16:27,158::task::595::Storage.TaskManager.Task::(_updateState) Task=`51af7346-2608-4f42-8a0e-6594b38b76c2`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:16:27,158::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:16:27,159::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:16:27,159::task::993::Storage.TaskManager.Task::(_decref) Task=`51af7346-2608-4f42-8a0e-6594b38b76c2`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:16:27,171::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:16:28,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:28,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915101 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:30,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:31,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543907 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:16:31,935::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:16:33,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:33,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546035 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:16:35,266::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:16:35,267::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:16:35,668::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:16:35,679::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:16:35,691::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000695198 s, 534 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:16:35,692::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00079285 s, 660 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:35,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:35,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914444 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:37,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:38,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915362 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:40,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:40,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0934657 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:42,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:16:42,264::task::595::Storage.TaskManager.Task::(_updateState) Task=`52fe5175-ac40-4fd1-8eb7-ed75ccd067bb`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:16:42,264::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:16:42,265::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000695198', 'lastCheck': '6.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00079285', 'lastCheck': '6.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:16:42,265::task::1191::Storage.TaskManager.Task::(prepare) Task=`52fe5175-ac40-4fd1-8eb7-ed75ccd067bb`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000695198', 'lastCheck': '6.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00079285', 'lastCheck': '6.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:16:42,265::task::595::Storage.TaskManager.Task::(_updateState) Task=`52fe5175-ac40-4fd1-8eb7-ed75ccd067bb`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:16:42,265::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:16:42,265::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:16:42,266::task::993::Storage.TaskManager.Task::(_decref) Task=`52fe5175-ac40-4fd1-8eb7-ed75ccd067bb`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:16:42,285::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:16:42,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915391 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:44,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:44,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544726 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:16:45,278::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:16:45,280::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:16:45,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:16:45,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:16:45,727::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000715701 s, 518 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:16:45,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000799548 s, 654 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:46,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:16:46,949::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:16:47,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915412 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:49,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:49,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09133 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:16:49,750::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:16:51,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:51,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913193 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:16:53,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:53,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914455 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:16:55,289::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:16:55,290::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:16:55,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:55,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912473 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:16:55,736::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:16:55,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:16:55,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667757 s, 783 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:16:55,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656329 s, 565 kB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:16:57,408::task::595::Storage.TaskManager.Task::(_updateState) Task=`ddbb08b4-8c54-42cc-b54d-453190a7d1c2`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:16:57,409::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:16:57,409::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656329', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667757', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:16:57,409::task::1191::Storage.TaskManager.Task::(prepare) Task=`ddbb08b4-8c54-42cc-b54d-453190a7d1c2`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656329', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667757', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:16:57,409::task::595::Storage.TaskManager.Task::(_updateState) Task=`ddbb08b4-8c54-42cc-b54d-453190a7d1c2`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:16:57,409::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:16:57,410::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:16:57,410::task::993::Storage.TaskManager.Task::(_decref) Task=`ddbb08b4-8c54-42cc-b54d-453190a7d1c2`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:16:57,421::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:16:57,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:16:58,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560934 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:00,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:00,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915053 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:17:01,952::task::595::Storage.TaskManager.Task::(_updateState) Task=`dbf59b5a-096c-43ca-8bbd-4c7f47e154b1`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:17:01,952::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:17:01,955::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:17:01,955::task::1191::Storage.TaskManager.Task::(prepare) Task=`dbf59b5a-096c-43ca-8bbd-4c7f47e154b1`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:17:01,955::task::595::Storage.TaskManager.Task::(_updateState) Task=`dbf59b5a-096c-43ca-8bbd-4c7f47e154b1`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:17:01,956::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:17:01,956::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:17:01,956::task::993::Storage.TaskManager.Task::(_decref) Task=`dbf59b5a-096c-43ca-8bbd-4c7f47e154b1`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:17:01,983::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:17:02,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:02,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914506 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:04,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:04,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555643 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:17:05,301::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:17:05,302::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:17:05,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:17:05,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:17:05,787::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000729244 s, 509 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:17:05,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000758753 s, 689 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:06,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:06,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915686 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:09,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:09,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915017 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:11,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:11,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916439 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:17:12,512::task::595::Storage.TaskManager.Task::(_updateState) Task=`30012ea2-bdb0-4231-90e0-db2bf15d6375`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:17:12,512::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:17:12,512::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000729244', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000758753', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:17:12,512::task::1191::Storage.TaskManager.Task::(prepare) Task=`30012ea2-bdb0-4231-90e0-db2bf15d6375`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000729244', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000758753', 'lastCheck': '6.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:17:12,513::task::595::Storage.TaskManager.Task::(_updateState) Task=`30012ea2-bdb0-4231-90e0-db2bf15d6375`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:17:12,513::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:17:12,513::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:17:12,513::task::993::Storage.TaskManager.Task::(_decref) Task=`30012ea2-bdb0-4231-90e0-db2bf15d6375`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:17:12,525::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:17:13,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:13,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923991 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:17:15,313::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:17:15,314::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:17:15,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:15,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916333 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:17:15,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:17:15,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:17:15,827::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684471 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:17:15,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000628774 s, 832 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:17:16,995::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:17:17,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:17,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091271 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:19,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:19,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920078 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:17:19,787::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:17:21,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:21,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:23,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:23,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913324 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:17:25,325::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:17:25,326::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:17:25,839::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:17:25,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:17:25,862::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00072183 s, 514 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:17:25,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676281 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:25,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:26,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912765 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:17:27,602::task::595::Storage.TaskManager.Task::(_updateState) Task=`3aaead28-3660-466a-938a-90f458cc541d`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:17:27,602::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:17:27,602::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00072183', 'lastCheck': '1.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000676281', 'lastCheck': '1.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:17:27,603::task::1191::Storage.TaskManager.Task::(prepare) Task=`3aaead28-3660-466a-938a-90f458cc541d`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00072183', 'lastCheck': '1.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000676281', 'lastCheck': '1.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:17:27,603::task::595::Storage.TaskManager.Task::(_updateState) Task=`3aaead28-3660-466a-938a-90f458cc541d`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:17:27,603::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:17:27,603::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:17:27,603::task::993::Storage.TaskManager.Task::(_decref) Task=`3aaead28-3660-466a-938a-90f458cc541d`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:17:27,614::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:17:28,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:28,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912838 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:30,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:30,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913763 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:17:32,007::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:17:32,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:32,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915204 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:34,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:34,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558702 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:17:35,338::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:17:35,339::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 15:17:35,867::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:17:35,867::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:17:35,882::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000800462 s, 463 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:17:35,896::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:17:35,897::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:17:35,900::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:17:35,901::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:17:35,902::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:17:35,902::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:17:35,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:17:35,914::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666137 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:37,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:37,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544737 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:39,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:39,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914078 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:41,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:41,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914451 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:17:42,699::task::595::Storage.TaskManager.Task::(_updateState) Task=`6856f523-f129-40b7-8050-aeff266120da`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:17:42,699::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:17:42,700::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000800462', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000666137', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:17:42,700::task::1191::Storage.TaskManager.Task::(prepare) Task=`6856f523-f129-40b7-8050-aeff266120da`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000800462', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000666137', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:17:42,700::task::595::Storage.TaskManager.Task::(_updateState) Task=`6856f523-f129-40b7-8050-aeff266120da`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:17:42,701::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:17:42,701::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:17:42,701::task::993::Storage.TaskManager.Task::(_decref) Task=`6856f523-f129-40b7-8050-aeff266120da`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:17:42,712::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:17:43,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:43,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914854 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:17:45,350::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:17:45,351::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:17:45,891::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:17:45,905::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645684 s, 575 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:17:45,919::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:17:45,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00254874 s, 205 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:45,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:46,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.099055 s, 10.3 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:17:47,020::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:17:48,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:48,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913399 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:17:49,818::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:17:50,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:50,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911053 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:52,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:52,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556969 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:54,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:55,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912974 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:17:55,364::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:17:55,366::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:17:55,911::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:17:55,923::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00060315 s, 615 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:17:55,951::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:17:55,962::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679694 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:17:57,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:17:57,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558273 s, 1.8 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:17:57,828::task::595::Storage.TaskManager.Task::(_updateState) Task=`83e6b879-42de-41c6-911d-88bfd2fd9d20`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:17:57,829::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:17:57,829::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00060315', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679694', 'lastCheck': '1.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:17:57,829::task::1191::Storage.TaskManager.Task::(prepare) Task=`83e6b879-42de-41c6-911d-88bfd2fd9d20`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00060315', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000679694', 'lastCheck': '1.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:17:57,829::task::595::Storage.TaskManager.Task::(_updateState) Task=`83e6b879-42de-41c6-911d-88bfd2fd9d20`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:17:57,830::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:17:57,830::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:17:57,830::task::993::Storage.TaskManager.Task::(_decref) Task=`83e6b879-42de-41c6-911d-88bfd2fd9d20`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:17:57,842::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:17:59,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:00,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551567 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:18:02,023::task::595::Storage.TaskManager.Task::(_updateState) Task=`12ed21d3-789c-44d8-a053-08b963a27efe`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:18:02,024::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:18:02,027::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:18:02,027::task::1191::Storage.TaskManager.Task::(prepare) Task=`12ed21d3-789c-44d8-a053-08b963a27efe`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:18:02,027::task::595::Storage.TaskManager.Task::(_updateState) Task=`12ed21d3-789c-44d8-a053-08b963a27efe`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:18:02,028::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:18:02,028::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:18:02,028::task::993::Storage.TaskManager.Task::(_decref) Task=`12ed21d3-789c-44d8-a053-08b963a27efe`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:18:02,054::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:18:02,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:02,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543621 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:04,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:04,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914924 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:18:05,377::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:18:05,378::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:18:05,925::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:18:05,939::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:18:05,941::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:18:05,944::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:18:05,945::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:18:05,945::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:18:05,945::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:18:05,947::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:18:05,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686293 s, 541 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:18:05,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:18:05,978::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647369 s, 808 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:06,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:07,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915546 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:09,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:09,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550964 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:11,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:11,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913192 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:18:12,945::task::595::Storage.TaskManager.Task::(_updateState) Task=`647d5d9a-421f-48e3-9b95-34be603f22dd`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:18:12,945::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:18:12,945::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000686293', 'lastCheck': '7.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647369', 'lastCheck': '7.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:18:12,945::task::1191::Storage.TaskManager.Task::(prepare) Task=`647d5d9a-421f-48e3-9b95-34be603f22dd`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000686293', 'lastCheck': '7.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647369', 'lastCheck': '7.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:18:12,946::task::595::Storage.TaskManager.Task::(_updateState) Task=`647d5d9a-421f-48e3-9b95-34be603f22dd`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:18:12,946::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:18:12,946::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:18:12,946::task::993::Storage.TaskManager.Task::(_decref) Task=`647d5d9a-421f-48e3-9b95-34be603f22dd`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:18:12,972::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:18:13,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:13,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916787 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:18:15,384::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:18:15,385::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:18:15,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:18:15,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:18:15,989::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:18:16,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0033822 s, 155 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:18:16,016::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674121 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:16,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.142905 s, 7.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:18:17,066::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:18:18,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:18,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914123 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:18:19,851::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:18:20,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:20,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913588 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:22,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:22,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914485 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:24,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:24,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913214 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:18:25,394::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:18:25,395::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:18:26,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:18:26,040::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652239 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:18:26,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:18:26,054::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693081 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:26,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:26,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.105929 s, 9.7 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:18:28,066::task::595::Storage.TaskManager.Task::(_updateState) Task=`1264da3a-a6ac-49ea-b753-50ee6916d314`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:18:28,066::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:18:28,067::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000652239', 'lastCheck': '2.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000693081', 'lastCheck': '2.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:18:28,067::task::1191::Storage.TaskManager.Task::(prepare) Task=`1264da3a-a6ac-49ea-b753-50ee6916d314`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000652239', 'lastCheck': '2.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000693081', 'lastCheck': '2.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:18:28,067::task::595::Storage.TaskManager.Task::(_updateState) Task=`1264da3a-a6ac-49ea-b753-50ee6916d314`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:18:28,067::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:18:28,068::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:18:28,068::task::993::Storage.TaskManager.Task::(_decref) Task=`1264da3a-a6ac-49ea-b753-50ee6916d314`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:18:28,080::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:18:28,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:28,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914156 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:30,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:31,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920483 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:18:32,079::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:18:33,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:33,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542959 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:18:35,406::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:18:35,407::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:18:35,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:18:36,204::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:18:36,215::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:18:36,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000732342 s, 714 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:18:36,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00076587 s, 484 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:36,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.582077 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:38,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:38,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558319 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:40,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:41,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914097 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:43,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:18:43,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`24cdf700-5d4e-4858-8755-d3300b63a650`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:18:43,174::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:18:43,175::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00076587', 'lastCheck': '6.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000732342', 'lastCheck': '6.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:18:43,175::task::1191::Storage.TaskManager.Task::(prepare) Task=`24cdf700-5d4e-4858-8755-d3300b63a650`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00076587', 'lastCheck': '6.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000732342', 'lastCheck': '6.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:18:43,175::task::595::Storage.TaskManager.Task::(_updateState) Task=`24cdf700-5d4e-4858-8755-d3300b63a650`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:18:43,175::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:18:43,176::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:18:43,176::task::993::Storage.TaskManager.Task::(_decref) Task=`24cdf700-5d4e-4858-8755-d3300b63a650`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:18:43,187::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:18:43,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543255 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:18:45,414::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:18:45,415::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:18:45,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:46,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546023 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:18:46,244::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:18:46,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:18:46,273::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000915625 s, 571 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:18:46,273::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684182 s, 542 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:18:47,091::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:18:48,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:48,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091314 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:18:49,884::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:18:50,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:51,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550656 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:53,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:53,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912851 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:55,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:55,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913563 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:18:55,427::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:18:55,428::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:18:56,284::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:18:56,296::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:18:56,307::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00424522 s, 87.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:18:56,308::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00082402 s, 635 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:18:57,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:18:57,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917761 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:18:58,310::task::595::Storage.TaskManager.Task::(_updateState) Task=`12e5edd1-bff6-4c0f-a341-20d30a31b778`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:18:58,311::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:18:58,311::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00424522', 'lastCheck': '2.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00082402', 'lastCheck': '2.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:18:58,311::task::1191::Storage.TaskManager.Task::(prepare) Task=`12e5edd1-bff6-4c0f-a341-20d30a31b778`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00424522', 'lastCheck': '2.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00082402', 'lastCheck': '2.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:18:58,311::task::595::Storage.TaskManager.Task::(_updateState) Task=`12e5edd1-bff6-4c0f-a341-20d30a31b778`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:18:58,312::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:18:58,312::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:18:58,312::task::993::Storage.TaskManager.Task::(_decref) Task=`12e5edd1-bff6-4c0f-a341-20d30a31b778`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:18:58,323::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:18:59,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:00,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556627 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:02,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:19:02,094::task::595::Storage.TaskManager.Task::(_updateState) Task=`08c0f258-b655-444d-91fa-193e834157f6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:19:02,095::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:19:02,128::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:19:02,129::task::1191::Storage.TaskManager.Task::(prepare) Task=`08c0f258-b655-444d-91fa-193e834157f6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:19:02,130::task::595::Storage.TaskManager.Task::(_updateState) Task=`08c0f258-b655-444d-91fa-193e834157f6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:19:02,131::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:19:02,131::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:19:02,132::task::993::Storage.TaskManager.Task::(_decref) Task=`08c0f258-b655-444d-91fa-193e834157f6`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 15:19:02,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964029 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:19:02,160::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:19:04,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:04,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911736 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:19:05,439::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:19:05,440::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:19:06,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:19:06,370::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:19:06,370::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:19:06,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00515263 s, 72.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:19:06,397::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662266 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:06,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129954 s, 7.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:08,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:08,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091304 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:10,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:10,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914465 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:12,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:12,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912927 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:19:13,422::task::595::Storage.TaskManager.Task::(_updateState) Task=`41073905-4cfe-4ebb-b764-202558753c7f`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:19:13,422::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:19:13,423::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00515263', 'lastCheck': '7.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000662266', 'lastCheck': '7.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:19:13,423::task::1191::Storage.TaskManager.Task::(prepare) Task=`41073905-4cfe-4ebb-b764-202558753c7f`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00515263', 'lastCheck': '7.0', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000662266', 'lastCheck': '7.0', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:19:13,423::task::595::Storage.TaskManager.Task::(_updateState) Task=`41073905-4cfe-4ebb-b764-202558753c7f`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:19:13,423::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:19:13,424::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:19:13,424::task::993::Storage.TaskManager.Task::(_decref) Task=`41073905-4cfe-4ebb-b764-202558753c7f`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:19:13,437::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:19:14,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:14,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914632 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:19:15,447::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:19:15,448::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:19:16,406::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:19:16,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642022 s, 578 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:19:16,424::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:19:16,437::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678329 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:16,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:16,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912603 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:19:17,172::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:19:19,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:19,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917392 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:19:19,919::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:19:21,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:21,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915796 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:23,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:23,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558673 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:19:25,456::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:19:25,457::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:19:25,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:25,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913077 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:19:26,427::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:19:26,438::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697135 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:19:26,445::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:19:26,456::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065472 s, 799 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:27,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:28,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913074 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:19:28,531::task::595::Storage.TaskManager.Task::(_updateState) Task=`577eb4f8-23ec-4b6a-8390-e58318b9eb6a`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:19:28,532::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:19:28,532::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000697135', 'lastCheck': '2.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00065472', 'lastCheck': '2.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:19:28,532::task::1191::Storage.TaskManager.Task::(prepare) Task=`577eb4f8-23ec-4b6a-8390-e58318b9eb6a`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000697135', 'lastCheck': '2.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00065472', 'lastCheck': '2.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:19:28,532::task::595::Storage.TaskManager.Task::(_updateState) Task=`577eb4f8-23ec-4b6a-8390-e58318b9eb6a`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:19:28,533::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:19:28,533::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:19:28,533::task::993::Storage.TaskManager.Task::(_decref) Task=`577eb4f8-23ec-4b6a-8390-e58318b9eb6a`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:19:28,544::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:19:30,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:30,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913301 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:32,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:19:32,196::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:19:32,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914397 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:34,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:34,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914856 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:19:35,468::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:19:35,469::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:19:36,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:19:36,520::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:19:36,521::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:19:36,534::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672425 s, 552 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:19:36,549::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00060142 s, 870 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:36,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.130169 s, 7.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:38,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:39,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559838 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:41,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:41,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915768 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:43,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:43,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924417 s, 11.1 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:19:43,632::task::595::Storage.TaskManager.Task::(_updateState) Task=`cbf74ba2-6b0c-48f5-8416-13ca84b18848`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:19:43,633::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:19:43,633::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000672425', 'lastCheck': '7.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00060142', 'lastCheck': '7.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:19:43,633::task::1191::Storage.TaskManager.Task::(prepare) Task=`cbf74ba2-6b0c-48f5-8416-13ca84b18848`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000672425', 'lastCheck': '7.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00060142', 'lastCheck': '7.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:19:43,634::task::595::Storage.TaskManager.Task::(_updateState) Task=`cbf74ba2-6b0c-48f5-8416-13ca84b18848`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:19:43,634::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:19:43,634::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:19:43,634::task::993::Storage.TaskManager.Task::(_decref) Task=`cbf74ba2-6b0c-48f5-8416-13ca84b18848`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:19:43,646::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:19:45,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:19:45,477::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:19:45,478::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:19:45,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918005 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:19:46,556::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:19:46,567::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:19:46,579::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000597185 s, 621 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:19:46,581::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690684 s, 757 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:19:47,209::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:19:47,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:47,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915074 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:49,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:49,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916444 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:19:49,952::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:19:51,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:51,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913567 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:53,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:53,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916296 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:19:55,489::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:19:55,490::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:19:55,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:56,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132989 s, 7.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:19:56,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:19:56,596::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:19:56,611::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000708413 s, 524 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:19:56,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000970115 s, 539 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:19:58,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:19:58,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558246 s, 1.8 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:19:58,746::task::595::Storage.TaskManager.Task::(_updateState) Task=`c145af7d-7946-4e0b-a53b-e19c37661429`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:19:58,746::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:19:58,746::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000708413', 'lastCheck': '2.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000970115', 'lastCheck': '2.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:19:58,747::task::1191::Storage.TaskManager.Task::(prepare) Task=`c145af7d-7946-4e0b-a53b-e19c37661429`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000708413', 'lastCheck': '2.1', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000970115', 'lastCheck': '2.1', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:19:58,747::task::595::Storage.TaskManager.Task::(_updateState) Task=`c145af7d-7946-4e0b-a53b-e19c37661429`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:19:58,747::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:19:58,747::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:19:58,747::task::993::Storage.TaskManager.Task::(_decref) Task=`c145af7d-7946-4e0b-a53b-e19c37661429`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:19:58,759::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:20:00,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:00,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912392 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:20:02,212::task::595::Storage.TaskManager.Task::(_updateState) Task=`d58f1222-0bd0-4f8f-ba88-62dcff135526`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:20:02,213::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:20:02,216::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:20:02,216::task::1191::Storage.TaskManager.Task::(prepare) Task=`d58f1222-0bd0-4f8f-ba88-62dcff135526`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:20:02,216::task::595::Storage.TaskManager.Task::(_updateState) Task=`d58f1222-0bd0-4f8f-ba88-62dcff135526`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:20:02,216::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:20:02,216::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:20:02,217::task::993::Storage.TaskManager.Task::(_decref) Task=`d58f1222-0bd0-4f8f-ba88-62dcff135526`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:20:02,243::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:20:02,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:02,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917224 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:04,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:05,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914207 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:20:05,497::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:20:05,498::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:20:06,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:20:06,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:20:06,640::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000740772 s, 501 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:20:06,643::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000819579 s, 638 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:07,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:08,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.880014 s, 1.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:10,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:10,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0943131 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:12,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:12,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.562239 s, 1.8 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:20:13,852::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3a3aafb-93ba-4fb8-9bdc-55ff7de053e4`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:20:13,852::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:20:13,852::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000740772', 'lastCheck': '7.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000819579', 'lastCheck': '7.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:20:13,852::task::1191::Storage.TaskManager.Task::(prepare) Task=`e3a3aafb-93ba-4fb8-9bdc-55ff7de053e4`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000740772', 'lastCheck': '7.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000819579', 'lastCheck': '7.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:20:13,852::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3a3aafb-93ba-4fb8-9bdc-55ff7de053e4`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:20:13,853::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:20:13,853::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:20:13,853::task::993::Storage.TaskManager.Task::(_decref) Task=`e3a3aafb-93ba-4fb8-9bdc-55ff7de053e4`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:20:13,864::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:20:14,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:14,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914678 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:20:15,509::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:20:15,510::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:20:16,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:20:16,658::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:20:16,669::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00275083 s, 135 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:20:16,674::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643387 s, 813 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:16,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:16,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914856 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:20:17,256::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:20:18,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:19,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926237 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:20:19,984::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:20:21,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:21,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:23,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:23,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918254 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:25,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:20:25,517::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:20:25,518::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:20:25,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544657 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:20:26,686::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:20:26,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000616848 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:20:26,703::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:20:26,715::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000736476 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:27,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:28,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917623 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:20:28,941::task::595::Storage.TaskManager.Task::(_updateState) Task=`a95be46c-8e79-47e0-9af1-35730f66ea3d`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:20:28,941::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:20:28,941::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000616848', 'lastCheck': '2.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000736476', 'lastCheck': '2.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:20:28,941::task::1191::Storage.TaskManager.Task::(prepare) Task=`a95be46c-8e79-47e0-9af1-35730f66ea3d`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000616848', 'lastCheck': '2.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000736476', 'lastCheck': '2.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:20:28,942::task::595::Storage.TaskManager.Task::(_updateState) Task=`a95be46c-8e79-47e0-9af1-35730f66ea3d`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:20:28,942::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:20:28,942::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:20:28,942::task::993::Storage.TaskManager.Task::(_decref) Task=`a95be46c-8e79-47e0-9af1-35730f66ea3d`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:20:28,953::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:20:30,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:30,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914593 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:32,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:20:32,270::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:20:32,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914159 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:34,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:34,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913877 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:20:35,529::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:20:35,530::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:20:36,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:36,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913469 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:20:36,717::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:20:36,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:20:36,739::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069469 s, 534 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:20:36,742::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000753409 s, 694 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:38,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:38,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915632 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:40,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:40,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915948 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:42,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:42,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913667 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:20:44,071::task::595::Storage.TaskManager.Task::(_updateState) Task=`bae7fdca-d045-44bd-80cc-01c90c090a7e`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:20:44,072::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:20:44,072::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00069469', 'lastCheck': '7.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000753409', 'lastCheck': '7.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:20:44,072::task::1191::Storage.TaskManager.Task::(prepare) Task=`bae7fdca-d045-44bd-80cc-01c90c090a7e`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00069469', 'lastCheck': '7.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000753409', 'lastCheck': '7.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:20:44,072::task::595::Storage.TaskManager.Task::(_updateState) Task=`bae7fdca-d045-44bd-80cc-01c90c090a7e`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:20:44,073::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:20:44,073::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:20:44,073::task::993::Storage.TaskManager.Task::(_decref) Task=`bae7fdca-d045-44bd-80cc-01c90c090a7e`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:20:44,089::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:20:44,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:45,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918157 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:20:45,541::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:20:45,542::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:20:46,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:20:46,755::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:20:46,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620903 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:20:46,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000628737 s, 832 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:47,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:20:47,282::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:20:47,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55857 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:49,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:49,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0935824 s, 10.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:20:50,026::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:20:51,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:51,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922515 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:53,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:54,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545317 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:20:55,553::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:20:55,554::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:20:56,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:56,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913957 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:20:56,775::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:20:56,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:20:56,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00167318 s, 222 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:20:56,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00117382 s, 446 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:20:58,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:20:58,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915641 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:20:59,183::task::595::Storage.TaskManager.Task::(_updateState) Task=`7b7ab8eb-2466-47c9-b60f-a2cf31c997da`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:20:59,183::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:20:59,183::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00167318', 'lastCheck': '2.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00117382', 'lastCheck': '2.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:20:59,184::task::1191::Storage.TaskManager.Task::(prepare) Task=`7b7ab8eb-2466-47c9-b60f-a2cf31c997da`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00167318', 'lastCheck': '2.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00117382', 'lastCheck': '2.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:20:59,184::task::595::Storage.TaskManager.Task::(_updateState) Task=`7b7ab8eb-2466-47c9-b60f-a2cf31c997da`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:20:59,184::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:20:59,184::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:20:59,184::task::993::Storage.TaskManager.Task::(_decref) Task=`7b7ab8eb-2466-47c9-b60f-a2cf31c997da`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:20:59,202::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:21:00,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:00,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915305 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:21:02,285::task::595::Storage.TaskManager.Task::(_updateState) Task=`650c294d-fe7a-4ca5-9e8b-9f72c00b495d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:21:02,286::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:21:02,288::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:21:02,288::task::1191::Storage.TaskManager.Task::(prepare) Task=`650c294d-fe7a-4ca5-9e8b-9f72c00b495d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:21:02,288::task::595::Storage.TaskManager.Task::(_updateState) Task=`650c294d-fe7a-4ca5-9e8b-9f72c00b495d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:21:02,289::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:21:02,289::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:21:02,289::task::993::Storage.TaskManager.Task::(_decref) Task=`650c294d-fe7a-4ca5-9e8b-9f72c00b495d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:21:02,317::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:21:02,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:02,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915719 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:05,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:05,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917879 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:21:05,565::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:21:05,567::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:21:07,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:21:07,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:21:07,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00291579 s, 127 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:21:07,115::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692657 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:07,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:07,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091536 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:09,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:09,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091529 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:11,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:11,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914487 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:13,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:13,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914388 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:21:14,302::task::595::Storage.TaskManager.Task::(_updateState) Task=`054c3be8-a878-4c20-a3c1-51446d1c8534`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:21:14,303::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:21:14,303::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00291579', 'lastCheck': '7.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000692657', 'lastCheck': '7.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:21:14,303::task::1191::Storage.TaskManager.Task::(prepare) Task=`054c3be8-a878-4c20-a3c1-51446d1c8534`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00291579', 'lastCheck': '7.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000692657', 'lastCheck': '7.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:21:14,303::task::595::Storage.TaskManager.Task::(_updateState) Task=`054c3be8-a878-4c20-a3c1-51446d1c8534`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:21:14,304::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:21:14,304::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:21:14,304::task::993::Storage.TaskManager.Task::(_decref) Task=`054c3be8-a878-4c20-a3c1-51446d1c8534`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:21:14,316::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:21:15,580::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:21:15,583::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:21:15,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:16,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560591 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:21:17,117::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:21:17,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:21:17,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661246 s, 561 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:21:17,142::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685294 s, 763 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:21:17,330::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:21:18,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:18,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913069 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:21:20,059::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:21:20,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:20,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:22,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:23,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545385 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:25,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:25,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915083 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:21:25,586::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:21:25,587::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:21:27,147::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:21:27,158::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:21:27,169::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000748225 s, 496 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:21:27,171::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687244 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:27,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:27,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914035 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:29,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:21:29,396::task::595::Storage.TaskManager.Task::(_updateState) Task=`4914428d-ee47-4a7e-96df-51c1a161c01a`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:21:29,396::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:21:29,397::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000748225', 'lastCheck': '2.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000687244', 'lastCheck': '2.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:21:29,397::task::1191::Storage.TaskManager.Task::(prepare) Task=`4914428d-ee47-4a7e-96df-51c1a161c01a`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000748225', 'lastCheck': '2.2', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000687244', 'lastCheck': '2.2', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:21:29,397::task::595::Storage.TaskManager.Task::(_updateState) Task=`4914428d-ee47-4a7e-96df-51c1a161c01a`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:21:29,397::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:21:29,397::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:21:29,397::task::993::Storage.TaskManager.Task::(_decref) Task=`4914428d-ee47-4a7e-96df-51c1a161c01a`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:21:29,416::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:21:29,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915197 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:31,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:31,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0934444 s, 11.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:21:32,343::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:21:33,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:33,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914666 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:21:35,598::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:21:35,599::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:21:35,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:35,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917373 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:21:37,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:21:37,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:21:37,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0033487 s, 111 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:21:37,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000654798 s, 799 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:37,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:38,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545004 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:40,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:40,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914852 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:42,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:42,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091478 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:21:44,498::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a8e7fab-6171-4ae8-a348-5b36753b57fa`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:21:44,498::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:21:44,498::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0033487', 'lastCheck': '7.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000654798', 'lastCheck': '7.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:21:44,499::task::1191::Storage.TaskManager.Task::(prepare) Task=`4a8e7fab-6171-4ae8-a348-5b36753b57fa`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0033487', 'lastCheck': '7.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000654798', 'lastCheck': '7.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:21:44,499::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a8e7fab-6171-4ae8-a348-5b36753b57fa`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:21:44,499::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:21:44,499::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:21:44,499::task::993::Storage.TaskManager.Task::(_decref) Task=`4a8e7fab-6171-4ae8-a348-5b36753b57fa`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:21:44,511::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:21:44,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:21:45,613::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:21:45,617::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:21:46,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38722 s, 738 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:21:47,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:21:47,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:21:47,228::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626765 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:21:47,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000897919 s, 582 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:21:47,356::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:21:48,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:48,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911853 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:21:50,093::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:21:50,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:50,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921154 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:52,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:53,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544015 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:55,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:55,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914742 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:21:55,630::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:21:55,634::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:21:57,234::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:21:57,246::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:21:57,257::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00535711 s, 69.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:21:57,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064222 s, 814 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:21:57,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:21:57,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920413 s, 11.1 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:21:59,600::task::595::Storage.TaskManager.Task::(_updateState) Task=`6fdcfe9a-01fe-4ab4-8745-8f665210f873`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:21:59,600::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:21:59,600::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00535711', 'lastCheck': '2.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00064222', 'lastCheck': '2.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:21:59,601::task::1191::Storage.TaskManager.Task::(prepare) Task=`6fdcfe9a-01fe-4ab4-8745-8f665210f873`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00535711', 'lastCheck': '2.3', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00064222', 'lastCheck': '2.3', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:21:59,601::task::595::Storage.TaskManager.Task::(_updateState) Task=`6fdcfe9a-01fe-4ab4-8745-8f665210f873`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:21:59,601::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:21:59,601::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:21:59,601::task::993::Storage.TaskManager.Task::(_decref) Task=`6fdcfe9a-01fe-4ab4-8745-8f665210f873`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:21:59,613::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:21:59,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:00,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543929 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:02,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:22:02,359::task::595::Storage.TaskManager.Task::(_updateState) Task=`055b9f93-b32b-46fd-9f26-4b4688d4ec02`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:22:02,359::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:22:02,437::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:22:02,438::task::1191::Storage.TaskManager.Task::(prepare) Task=`055b9f93-b32b-46fd-9f26-4b4688d4ec02`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:22:02,438::task::595::Storage.TaskManager.Task::(_updateState) Task=`055b9f93-b32b-46fd-9f26-4b4688d4ec02`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:22:02,439::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:22:02,440::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:22:02,441::task::993::Storage.TaskManager.Task::(_decref) Task=`055b9f93-b32b-46fd-9f26-4b4688d4ec02`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 15:22:02,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0961856 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:22:02,470::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:22:04,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:04,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912638 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:22:05,634::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:22:05,635::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:22:06,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:06,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915302 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:22:07,265::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:22:07,276::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:22:07,287::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000744787 s, 498 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:22:07,289::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000813716 s, 643 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:08,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:08,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914364 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:10,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:10,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914143 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:12,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:13,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914136 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:22:14,706::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c936e19-ce3c-43a1-9838-03326d9f7bb3`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:22:14,706::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:22:14,707::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000744787', 'lastCheck': '7.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000813716', 'lastCheck': '7.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:22:14,707::task::1191::Storage.TaskManager.Task::(prepare) Task=`1c936e19-ce3c-43a1-9838-03326d9f7bb3`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000744787', 'lastCheck': '7.4', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000813716', 'lastCheck': '7.4', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:22:14,707::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c936e19-ce3c-43a1-9838-03326d9f7bb3`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:22:14,707::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:22:14,707::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:22:14,708::task::993::Storage.TaskManager.Task::(_decref) Task=`1c936e19-ce3c-43a1-9838-03326d9f7bb3`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:22:14,719::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:22:15,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:15,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558841 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:22:15,648::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:22:15,654::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:22:17,293::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:22:17,304::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:22:17,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00436285 s, 85.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:22:17,317::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000828132 s, 632 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:22:17,483::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:22:17,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:17,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920076 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:19,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:19,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914041 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:22:20,126::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:22:22,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:23,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.3941 s, 735 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:25,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:25,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091493 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:22:25,656::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:22:25,657::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:22:27,319::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:22:27,331::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665817 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:22:27,332::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:22:27,344::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658208 s, 795 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:27,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:27,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913419 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:29,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:29,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913348 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:22:29,815::task::595::Storage.TaskManager.Task::(_updateState) Task=`a2d907fc-fa97-4061-9ed9-a01253756e3c`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:22:29,816::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:22:29,816::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000665817', 'lastCheck': '2.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000658208', 'lastCheck': '2.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:22:29,816::task::1191::Storage.TaskManager.Task::(prepare) Task=`a2d907fc-fa97-4061-9ed9-a01253756e3c`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000665817', 'lastCheck': '2.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000658208', 'lastCheck': '2.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:22:29,816::task::595::Storage.TaskManager.Task::(_updateState) Task=`a2d907fc-fa97-4061-9ed9-a01253756e3c`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:22:29,816::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:22:29,816::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:22:29,817::task::993::Storage.TaskManager.Task::(_decref) Task=`a2d907fc-fa97-4061-9ed9-a01253756e3c`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:22:29,842::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:22:31,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:31,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920165 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:22:32,496::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:22:33,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:33,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091362 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:22:35,668::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:22:35,669::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:22:36,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:36,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551143 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:22:37,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:22:37,355::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:22:37,361::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627895 s, 591 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:22:37,377::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:22:37,378::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:22:37,381::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:22:37,383::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:22:37,383::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:22:37,383::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:22:37,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:22:37,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650004 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:38,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:38,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:40,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:41,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546349 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:43,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:43,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552717 s, 1.9 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:22:44,918::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a59ad7e-58f7-4d82-b61c-43ec900584b0`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:22:44,919::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:22:44,919::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000627895', 'lastCheck': '7.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650004', 'lastCheck': '7.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:22:44,919::task::1191::Storage.TaskManager.Task::(prepare) Task=`4a59ad7e-58f7-4d82-b61c-43ec900584b0`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000627895', 'lastCheck': '7.5', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000650004', 'lastCheck': '7.5', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:22:44,919::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a59ad7e-58f7-4d82-b61c-43ec900584b0`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:22:44,919::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:22:44,919::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:22:44,920::task::993::Storage.TaskManager.Task::(_decref) Task=`4a59ad7e-58f7-4d82-b61c-43ec900584b0`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:22:44,931::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:22:45,683::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:22:45,687::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:22:45,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:46,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549526 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:22:47,373::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:22:47,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625294 s, 593 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:22:47,399::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:22:47,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656013 s, 797 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:22:47,509::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:22:48,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:48,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913412 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:22:50,160::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:22:50,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:51,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557706 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:53,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:53,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915264 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:55,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:55,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910228 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:22:55,693::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:22:55,694::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:22:57,388::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:22:57,400::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656698 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:22:57,415::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:22:57,427::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000609492 s, 858 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:57,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:57,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913109 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:22:59,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:22:59,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913442 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:23:00,004::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3fe81b7-248d-4fb5-bf45-eb5b3f166a3d`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:23:00,004::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:23:00,005::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656698', 'lastCheck': '2.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000609492', 'lastCheck': '2.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:23:00,005::task::1191::Storage.TaskManager.Task::(prepare) Task=`a3fe81b7-248d-4fb5-bf45-eb5b3f166a3d`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656698', 'lastCheck': '2.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000609492', 'lastCheck': '2.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:23:00,005::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3fe81b7-248d-4fb5-bf45-eb5b3f166a3d`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:23:00,005::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:23:00,005::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:23:00,005::task::993::Storage.TaskManager.Task::(_decref) Task=`a3fe81b7-248d-4fb5-bf45-eb5b3f166a3d`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:23:00,016::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:23:01,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:02,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558356 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:23:02,512::task::595::Storage.TaskManager.Task::(_updateState) Task=`59d7eb17-46bf-4b99-8cef-2840e5493be9`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:23:02,512::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:23:02,515::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:23:02,515::task::1191::Storage.TaskManager.Task::(prepare) Task=`59d7eb17-46bf-4b99-8cef-2840e5493be9`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:23:02,515::task::595::Storage.TaskManager.Task::(_updateState) Task=`59d7eb17-46bf-4b99-8cef-2840e5493be9`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:23:02,516::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:23:02,516::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:23:02,516::task::993::Storage.TaskManager.Task::(_decref) Task=`59d7eb17-46bf-4b99-8cef-2840e5493be9`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:23:02,543::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:23:04,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:04,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913836 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:23:05,705::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:23:05,706::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:23:06,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:06,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913859 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:23:07,402::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:23:07,415::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:23:07,416::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:23:07,419::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:23:07,421::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:23:07,421::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:23:07,422::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:23:07,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:23:07,435::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687424 s, 540 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:23:07,437::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:23:07,539::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687613 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:08,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:08,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914098 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:10,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:10,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091392 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:12,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:12,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913727 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:14,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:15,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919163 s, 11.1 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:23:15,108::task::595::Storage.TaskManager.Task::(_updateState) Task=`058302ff-07d6-41a6-b97f-acf841014d57`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:23:15,109::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:23:15,109::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000687424', 'lastCheck': '7.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000687613', 'lastCheck': '7.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:23:15,109::task::1191::Storage.TaskManager.Task::(prepare) Task=`058302ff-07d6-41a6-b97f-acf841014d57`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000687424', 'lastCheck': '7.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000687613', 'lastCheck': '7.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:23:15,109::task::595::Storage.TaskManager.Task::(_updateState) Task=`058302ff-07d6-41a6-b97f-acf841014d57`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:23:15,110::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:23:15,110::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:23:15,110::task::993::Storage.TaskManager.Task::(_decref) Task=`058302ff-07d6-41a6-b97f-acf841014d57`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:23:15,121::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:23:15,720::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:23:15,724::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:23:17,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:17,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915342 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:23:17,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:23:17,463::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654603 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:23:17,543::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:23:17,554::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672801 s, 777 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:23:17,566::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:23:19,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:19,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913957 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:23:20,193::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:23:21,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:21,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132491 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:23,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:23,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558546 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:23:25,730::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:23:25,731::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:23:25,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:26,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.119261 s, 8.6 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:23:27,466::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:23:27,477::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652552 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:23:27,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:23:27,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686942 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:28,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:28,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912636 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:23:30,200::task::595::Storage.TaskManager.Task::(_updateState) Task=`b29e439b-f7a8-4773-8e4c-52ce67c79433`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:23:30,200::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:23:30,200::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000652552', 'lastCheck': '2.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000686942', 'lastCheck': '2.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:23:30,200::task::1191::Storage.TaskManager.Task::(prepare) Task=`b29e439b-f7a8-4773-8e4c-52ce67c79433`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000652552', 'lastCheck': '2.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000686942', 'lastCheck': '2.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:23:30,201::task::595::Storage.TaskManager.Task::(_updateState) Task=`b29e439b-f7a8-4773-8e4c-52ce67c79433`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:23:30,201::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:23:30,201::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:23:30,201::task::993::Storage.TaskManager.Task::(_decref) Task=`b29e439b-f7a8-4773-8e4c-52ce67c79433`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:23:30,213::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:23:30,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:30,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913728 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:32,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:32,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914193 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:23:32,578::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:23:34,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:35,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54792 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:23:35,741::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:23:35,742::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:23:37,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:23:38,435::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:23:38,446::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:23:38,460::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000737791 s, 503 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:23:38,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676221 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:38,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.40043 s, 731 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:40,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:40,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915918 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:42,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:42,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915303 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:44,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:44,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915334 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:23:45,304::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9359ad0-fff0-4fe1-bb40-d9940fc12e0f`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:23:45,305::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:23:45,305::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000737791', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000676221', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:23:45,305::task::1191::Storage.TaskManager.Task::(prepare) Task=`e9359ad0-fff0-4fe1-bb40-d9940fc12e0f`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000737791', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000676221', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:23:45,305::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9359ad0-fff0-4fe1-bb40-d9940fc12e0f`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:23:45,305::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:23:45,306::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:23:45,306::task::993::Storage.TaskManager.Task::(_decref) Task=`e9359ad0-fff0-4fe1-bb40-d9940fc12e0f`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:23:45,317::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:23:45,757::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:23:45,761::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:23:46,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:46,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915337 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:23:47,590::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:23:48,471::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:23:48,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:23:48,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000616773 s, 602 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:23:48,497::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000625516 s, 836 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:48,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:49,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914592 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:23:50,226::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:23:51,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:51,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914689 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:53,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:53,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914577 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:55,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:55,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914233 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:23:55,768::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:23:55,769::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:23:57,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:57,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913498 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:23:58,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:23:58,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:23:58,525::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656662 s, 796 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:23:58,526::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00165713 s, 224 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:23:59,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:23:59,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913185 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:24:00,408::task::595::Storage.TaskManager.Task::(_updateState) Task=`f22b9a6e-45ea-49dc-abd2-952523edb50d`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:24:00,408::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:24:00,409::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00165713', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656662', 'lastCheck': '1.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:24:00,409::task::1191::Storage.TaskManager.Task::(prepare) Task=`f22b9a6e-45ea-49dc-abd2-952523edb50d`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00165713', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000656662', 'lastCheck': '1.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:24:00,409::task::595::Storage.TaskManager.Task::(_updateState) Task=`f22b9a6e-45ea-49dc-abd2-952523edb50d`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:24:00,409::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:24:00,409::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:24:00,410::task::993::Storage.TaskManager.Task::(_decref) Task=`f22b9a6e-45ea-49dc-abd2-952523edb50d`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:24:00,421::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:24:01,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:01,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.096196 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:24:02,593::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5bd7ede-e783-4a7c-ad5d-710c51d94430`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:24:02,593::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:24:02,597::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:24:02,597::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5bd7ede-e783-4a7c-ad5d-710c51d94430`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:24:02,597::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5bd7ede-e783-4a7c-ad5d-710c51d94430`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:24:02,597::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:24:02,597::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:24:02,597::task::993::Storage.TaskManager.Task::(_decref) Task=`a5bd7ede-e783-4a7c-ad5d-710c51d94430`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:24:02,623::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:24:03,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:03,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914111 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:24:05,778::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:24:05,779::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:24:05,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:06,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915788 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:08,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:24:08,595::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:24:08,606::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:24:08,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00564662 s, 92.6 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:24:08,621::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000764492 s, 485 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:08,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.588847 s, 1.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:10,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:10,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914215 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:12,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:12,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914244 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:14,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:24:15,510::task::595::Storage.TaskManager.Task::(_updateState) Task=`8dd169f7-16b8-4f83-9c7b-2a6c725aa0cf`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:24:15,510::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:24:15,510::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000764492', 'lastCheck': '6.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00564662', 'lastCheck': '6.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:24:15,510::task::1191::Storage.TaskManager.Task::(prepare) Task=`8dd169f7-16b8-4f83-9c7b-2a6c725aa0cf`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000764492', 'lastCheck': '6.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00564662', 'lastCheck': '6.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:24:15,511::task::595::Storage.TaskManager.Task::(_updateState) Task=`8dd169f7-16b8-4f83-9c7b-2a6c725aa0cf`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:24:15,511::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:24:15,511::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:24:15,511::task::993::Storage.TaskManager.Task::(_decref) Task=`8dd169f7-16b8-4f83-9c7b-2a6c725aa0cf`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:24:15,522::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:24:15,793::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:24:15,796::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:24:16,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38276 s, 741 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:24:17,636::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:24:18,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:24:18,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:24:18,978::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:24:18,979::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000703608 s, 527 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:24:18,996::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700864 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:19,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.59464 s, 1.7 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:24:20,258::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:24:21,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:21,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544494 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:23,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:23,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915218 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:25,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:24:25,798::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:24:25,800::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:24:25,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926578 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:27,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:27,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914507 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:24:29,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:24:29,014::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671102 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:24:29,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:24:29,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696518 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:29,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:30,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912086 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:24:30,604::task::595::Storage.TaskManager.Task::(_updateState) Task=`c55879a7-384e-4f93-8eb9-06c58db4f3ce`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:24:30,604::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:24:30,604::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000671102', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000696518', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:24:30,604::task::1191::Storage.TaskManager.Task::(prepare) Task=`c55879a7-384e-4f93-8eb9-06c58db4f3ce`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000671102', 'lastCheck': '1.6', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000696518', 'lastCheck': '1.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:24:30,605::task::595::Storage.TaskManager.Task::(_updateState) Task=`c55879a7-384e-4f93-8eb9-06c58db4f3ce`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:24:30,605::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:24:30,605::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:24:30,605::task::993::Storage.TaskManager.Task::(_decref) Task=`c55879a7-384e-4f93-8eb9-06c58db4f3ce`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:24:30,617::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:24:32,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:24:32,649::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:24:33,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550621 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:35,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:35,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915085 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:24:35,810::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:24:35,811::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:24:37,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:37,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914736 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:24:39,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:24:39,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:24:39,054::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000703394 s, 527 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:24:39,057::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00075254 s, 695 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:39,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:39,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915102 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:41,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:41,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913706 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:43,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:43,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913754 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:45,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:24:45,708::task::595::Storage.TaskManager.Task::(_updateState) Task=`a12370b1-4e4b-488d-b7fa-fa602d96093d`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:24:45,709::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:24:45,710::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000703394', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00075254', 'lastCheck': '6.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:24:45,711::task::1191::Storage.TaskManager.Task::(prepare) Task=`a12370b1-4e4b-488d-b7fa-fa602d96093d`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000703394', 'lastCheck': '6.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00075254', 'lastCheck': '6.6', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:24:45,711::task::595::Storage.TaskManager.Task::(_updateState) Task=`a12370b1-4e4b-488d-b7fa-fa602d96093d`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:24:45,712::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:24:45,712::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:24:45,713::task::993::Storage.TaskManager.Task::(_decref) Task=`a12370b1-4e4b-488d-b7fa-fa602d96093d`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:24:45,734::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:24:45,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.113943 s, 9.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:24:45,825::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:24:45,830::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:24:47,661::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:24:47,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:48,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.117373 s, 8.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:24:49,062::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:24:49,073::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:24:49,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000790916 s, 469 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:24:49,086::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665562 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:50,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:50,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913882 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:24:50,296::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:24:52,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:52,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913779 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:54,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:54,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964906 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:24:55,834::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:24:55,835::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:24:56,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:56,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915749 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:24:58,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:24:58,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915006 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:24:59,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:24:59,100::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:24:59,117::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068327 s, 543 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:24:59,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0012648 s, 414 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:00,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:00,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914225 s, 11.2 MB/s\n'; <rc> = 0 >Thread-11142::DEBUG::2014-07-14 15:25:00,825::task::595::Storage.TaskManager.Task::(_updateState) Task=`35e36902-346a-4afc-b574-7fdb8fda151e`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:25:00,825::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:25:00,826::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00068327', 'lastCheck': '1.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0012648', 'lastCheck': '1.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:25:00,826::task::1191::Storage.TaskManager.Task::(prepare) Task=`35e36902-346a-4afc-b574-7fdb8fda151e`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00068327', 'lastCheck': '1.7', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0012648', 'lastCheck': '1.7', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:25:00,826::task::595::Storage.TaskManager.Task::(_updateState) Task=`35e36902-346a-4afc-b574-7fdb8fda151e`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:25:00,826::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:25:00,826::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:25:00,826::task::993::Storage.TaskManager.Task::(_decref) Task=`35e36902-346a-4afc-b574-7fdb8fda151e`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:25:00,840::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:25:02,665::task::595::Storage.TaskManager.Task::(_updateState) Task=`85c333f9-67e3-456f-abc6-5240671b9c63`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:25:02,665::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:25:02,669::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:25:02,669::task::1191::Storage.TaskManager.Task::(prepare) Task=`85c333f9-67e3-456f-abc6-5240671b9c63`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:25:02,669::task::595::Storage.TaskManager.Task::(_updateState) Task=`85c333f9-67e3-456f-abc6-5240671b9c63`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:25:02,669::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:25:02,669::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:25:02,669::task::993::Storage.TaskManager.Task::(_decref) Task=`85c333f9-67e3-456f-abc6-5240671b9c63`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:25:02,697::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:25:02,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:02,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09175 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:04,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:04,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914981 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:25:05,847::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:25:05,848::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:25:06,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:07,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544685 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:25:09,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:25:09,133::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:25:09,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000761767 s, 487 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:25:09,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637477 s, 820 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:09,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:09,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913022 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:11,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:11,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913009 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:13,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:13,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914452 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:25:15,863::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:25:15,866::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:25:15,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:25:15,938::task::595::Storage.TaskManager.Task::(_updateState) Task=`8917d4e8-5f19-4f2b-8b74-c8e8660c2fe5`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:25:15,940::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:25:15,942::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000761767', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000637477', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:25:15,943::task::1191::Storage.TaskManager.Task::(prepare) Task=`8917d4e8-5f19-4f2b-8b74-c8e8660c2fe5`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000761767', 'lastCheck': '6.8', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000637477', 'lastCheck': '6.8', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:25:15,944::task::595::Storage.TaskManager.Task::(_updateState) Task=`8917d4e8-5f19-4f2b-8b74-c8e8660c2fe5`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:25:15,945::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:25:15,945::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:25:15,946::task::993::Storage.TaskManager.Task::(_decref) Task=`8917d4e8-5f19-4f2b-8b74-c8e8660c2fe5`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:25:15,977::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:25:16,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968113 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:25:17,708::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:25:18,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:18,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913869 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:25:19,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:25:19,162::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:25:19,172::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00466199 s, 79.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:25:19,175::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00063517 s, 823 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:20,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:25:20,329::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:25:20,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091464 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:22,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:22,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913712 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:24,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:24,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913573 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:25:25,869::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:25:25,870::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:25:26,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:26,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914766 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:28,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:28,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914132 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:25:29,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:25:29,188::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667483 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:25:29,190::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:25:29,204::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688166 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:30,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-11142::DEBUG::2014-07-14 15:25:31,077::task::595::Storage.TaskManager.Task::(_updateState) Task=`b30388c9-5fdf-4b52-bf38-968205d0002f`::moving from state init -> state preparing >Thread-11142::INFO::2014-07-14 15:25:31,077::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11142::INFO::2014-07-14 15:25:31,078::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667483', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000688166', 'lastCheck': '1.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:25:31,078::task::1191::Storage.TaskManager.Task::(prepare) Task=`b30388c9-5fdf-4b52-bf38-968205d0002f`::finished: {'f248ab94-c2d2-4d14-b120-b2fa7cc96393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000667483', 'lastCheck': '1.9', 'valid': True}, '1f0449ec-49a1-4c13-b973-a24382d1db5b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000688166', 'lastCheck': '1.9', 'valid': True}} >Thread-11142::DEBUG::2014-07-14 15:25:31,078::task::595::Storage.TaskManager.Task::(_updateState) Task=`b30388c9-5fdf-4b52-bf38-968205d0002f`::moving from state preparing -> state finished >Thread-11142::DEBUG::2014-07-14 15:25:31,078::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11142::DEBUG::2014-07-14 15:25:31,078::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11142::DEBUG::2014-07-14 15:25:31,079::task::993::Storage.TaskManager.Task::(_decref) Task=`b30388c9-5fdf-4b52-bf38-968205d0002f`::ref 0 aborting False >Thread-11142::ERROR::2014-07-14 15:25:31,091::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:25:31,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542229 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:25:32,719::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:25:33,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:33,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914187 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:35,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:35,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921134 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:25:35,881::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:25:35,882::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:25:37,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:37,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913712 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:25:39,205::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:25:39,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:25:39,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628182 s, 591 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:25:39,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000615814 s, 849 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:39,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:39,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091399 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:41,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:41,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913908 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:43,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:44,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914231 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:25:45,896::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:25:45,900::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:25:46,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:46,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968724 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:25:47,731::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:25:48,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:48,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913095 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:25:49,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:25:49,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:25:49,266::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006124 s, 606 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:25:49,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666816 s, 784 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:25:50,363::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:25:50,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:50,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912764 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:52,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:52,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913933 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:54,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:54,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925113 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:25:55,905::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:25:55,906::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:25:56,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:56,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912842 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:25:58,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:25:59,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912433 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:25:59,273::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:25:59,274::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:25:59,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00322766 s, 115 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:25:59,296::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681703 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:01,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:01,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919478 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:26:02,734::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b4985bd-afa0-4fb8-ba09-3f168f67d183`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:26:02,735::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:26:02,738::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:26:02,738::task::1191::Storage.TaskManager.Task::(prepare) Task=`0b4985bd-afa0-4fb8-ba09-3f168f67d183`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:26:02,738::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b4985bd-afa0-4fb8-ba09-3f168f67d183`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:26:02,738::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:26:02,738::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:26:02,739::task::993::Storage.TaskManager.Task::(_decref) Task=`0b4985bd-afa0-4fb8-ba09-3f168f67d183`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:26:02,766::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:26:03,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:03,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913754 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:05,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:05,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:26:05,915::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:26:05,916::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:26:07,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:07,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558742 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:26:09,301::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:26:09,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:26:09,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691294 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:26:09,327::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00075073 s, 697 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:09,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:10,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915487 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:12,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:12,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559007 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:14,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:14,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915326 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:26:15,931::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:26:15,936::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:26:16,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:16,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914039 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:26:17,778::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:26:18,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:19,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914351 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:26:19,328::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:26:19,340::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:26:19,350::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00603807 s, 61.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:26:19,351::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631212 s, 829 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:26:20,405::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:26:21,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:21,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.506971 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:23,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:23,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914052 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:25,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:25,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913816 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:26:25,941::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:26:25,942::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:26:27,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:27,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915029 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:26:29,354::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:26:29,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:26:29,367::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642716 s, 577 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:26:29,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000604058 s, 866 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:29,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:30,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914426 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:32,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:32,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557987 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:26:32,791::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:26:34,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:34,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914024 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:26:35,953::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:26:35,955::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:26:36,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:36,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914993 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:38,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:38,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922047 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:26:39,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:26:39,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:26:39,408::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000767184 s, 484 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:26:39,410::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698134 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:40,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:41,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913685 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:43,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:43,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.506345 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:45,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:45,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913273 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:26:45,969::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:26:45,973::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:26:47,804::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:26:47,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:47,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091391 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:26:49,413::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:26:49,425::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:26:49,442::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00059367 s, 625 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:26:49,444::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000617263 s, 847 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:49,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:50,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913125 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:26:50,442::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:26:52,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:52,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913984 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:54,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:54,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091335 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:26:55,979::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:26:55,980::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:26:56,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:56,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550823 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:26:58,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:26:59,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911673 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:26:59,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:26:59,462::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:26:59,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000770014 s, 679 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:26:59,475::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678194 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:01,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:01,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913478 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:27:02,806::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f14a275-1026-47fa-9297-7cf1315a12cd`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:27:02,807::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:27:02,809::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:27:02,810::task::1191::Storage.TaskManager.Task::(prepare) Task=`3f14a275-1026-47fa-9297-7cf1315a12cd`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:27:02,810::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f14a275-1026-47fa-9297-7cf1315a12cd`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:27:02,810::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:27:02,810::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:27:02,810::task::993::Storage.TaskManager.Task::(_decref) Task=`3f14a275-1026-47fa-9297-7cf1315a12cd`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:27:02,837::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:27:03,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:03,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915645 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:05,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:05,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541254 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:27:05,991::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:27:05,992::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:27:07,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:08,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913131 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:27:09,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:27:09,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:27:09,509::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678879 s, 546 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:27:09,511::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00109902 s, 476 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:10,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:10,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917921 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:12,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:12,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914825 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:14,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:14,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09156 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:27:16,013::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:27:16,017::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:27:16,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:16,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55706 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:27:17,850::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:27:18,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:19,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915304 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:27:19,589::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:27:19,599::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:27:19,612::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000722272 s, 724 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:27:19,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685337 s, 541 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:27:20,483::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:27:21,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:21,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914722 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:23,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:23,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913897 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:25,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:25,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.499087 s, 2.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:27:26,016::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:27:26,018::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:27:27,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:27,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914059 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:27:29,624::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:27:29,635::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:27:29,650::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00559978 s, 66.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:27:29,652::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675071 s, 775 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:29,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:30,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545601 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:32,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:32,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0931432 s, 11.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:27:32,862::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:27:34,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:34,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913885 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:27:36,029::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:27:36,030::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:27:36,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:36,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910387 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:38,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:38,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918026 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:27:39,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:27:39,675::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:27:39,677::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000624575 s, 594 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:27:39,706::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:27:39,708::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:27:39,711::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:27:39,712::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:27:39,713::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:27:39,713::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:27:39,714::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:27:39,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000605092 s, 864 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:40,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:41,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927481 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:43,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:43,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913586 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:45,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:45,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914168 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:27:46,045::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:27:46,051::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:27:47,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:47,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912304 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:27:47,875::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:27:49,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:49,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912881 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:27:49,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:27:49,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651892 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:27:49,732::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:27:49,746::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000705656 s, 741 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:27:50,513::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:27:51,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:51,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914663 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:53,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:53,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912864 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:27:55,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:55,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913429 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:27:56,052::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:27:56,054::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:27:57,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:27:58,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541882 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:27:59,703::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:27:59,714::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642792 s, 577 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:27:59,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:27:59,769::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068069 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:00,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:01,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545479 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:28:02,878::task::595::Storage.TaskManager.Task::(_updateState) Task=`fef2f829-0060-483a-b498-65f08688435d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:28:02,878::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:28:02,881::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:28:02,881::task::1191::Storage.TaskManager.Task::(prepare) Task=`fef2f829-0060-483a-b498-65f08688435d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:28:02,881::task::595::Storage.TaskManager.Task::(_updateState) Task=`fef2f829-0060-483a-b498-65f08688435d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:28:02,882::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:28:02,882::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:28:02,882::task::993::Storage.TaskManager.Task::(_decref) Task=`fef2f829-0060-483a-b498-65f08688435d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:28:02,910::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:28:03,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:03,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914445 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:05,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:05,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912681 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:28:06,060::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:28:06,061::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:28:07,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:07,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544912 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:28:09,716::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:28:09,739::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:28:09,741::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:28:09,744::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:28:09,745::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:28:09,745::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:28:09,746::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:28:09,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:28:09,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654197 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:28:09,773::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:28:09,785::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683213 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:09,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:09,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914262 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:11,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:12,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914234 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:14,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:14,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915244 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:28:16,071::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:28:16,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:28:16,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:16,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100353 s, 10.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:28:17,922::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:28:18,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:19,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543044 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:28:19,762::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:28:19,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671577 s, 552 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:28:19,789::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:28:19,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070039 s, 747 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:28:20,546::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:28:21,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:21,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913097 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:23,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:23,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914525 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:25,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:25,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548048 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:28:26,078::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:28:26,079::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:28:27,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:27,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916439 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:28:29,777::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:28:29,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660214 s, 562 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:28:29,804::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:28:29,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665315 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:29,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:30,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918212 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:32,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:32,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913761 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:28:32,934::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:28:34,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:34,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914259 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:28:36,088::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:28:36,089::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:28:36,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:36,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0948818 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:38,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:39,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556404 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:28:39,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:28:39,803::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670924 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:28:39,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:28:39,830::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663589 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:41,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:41,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091442 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:43,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:43,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551869 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:45,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:28:46,105::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:28:46,110::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:28:46,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548631 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:28:47,947::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:28:48,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:48,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0928348 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:28:49,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:28:49,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620569 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:28:49,833::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:28:49,845::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667598 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:50,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:28:50,579::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:28:51,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560794 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:53,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:53,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546673 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:55,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:55,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09189 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:28:56,114::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:28:56,115::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:28:57,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:28:57,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0954567 s, 10.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:28:59,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:28:59,833::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065022 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:28:59,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:28:59,860::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687048 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:28:59,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:00,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091308 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:02,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:02,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914046 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:29:02,950::task::595::Storage.TaskManager.Task::(_updateState) Task=`802e87a8-7fba-4095-8d2c-593816ec33b5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:29:02,951::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:29:02,953::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:29:02,953::task::1191::Storage.TaskManager.Task::(prepare) Task=`802e87a8-7fba-4095-8d2c-593816ec33b5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:29:02,953::task::595::Storage.TaskManager.Task::(_updateState) Task=`802e87a8-7fba-4095-8d2c-593816ec33b5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:29:02,954::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:29:02,954::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:29:02,954::task::993::Storage.TaskManager.Task::(_decref) Task=`802e87a8-7fba-4095-8d2c-593816ec33b5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:29:02,982::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:29:04,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:04,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914056 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:29:06,126::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:29:06,128::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:29:06,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:06,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914336 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:08,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:09,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561754 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:29:09,837::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:29:09,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693054 s, 535 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:29:09,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:29:09,875::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658749 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:11,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:11,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545909 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:13,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:13,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922964 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:15,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:15,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097294 s, 10.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:29:16,139::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:29:16,143::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:29:17,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:29:18,002::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:29:18,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924917 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:29:19,852::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:29:19,863::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680197 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:29:19,879::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:29:19,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684395 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:20,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:20,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091518 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:29:20,614::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:29:22,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:22,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914159 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:24,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:24,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914527 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:29:26,148::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:29:26,149::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:29:26,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:26,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911221 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:28,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:28,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913274 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:29:29,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:29:29,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684064 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:29:29,893::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:29:29,905::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679538 s, 770 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:30,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:30,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915136 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:32,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:32,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913915 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:29:33,013::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:29:34,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:34,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912414 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:29:36,160::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:29:36,162::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:29:37,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:37,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916356 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:39,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:29:40,534::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:29:40,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:40,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.42069 s, 721 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:29:40,623::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665983 s, 785 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:29:40,625::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0017173 s, 216 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:42,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:42,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911988 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:44,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:44,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914676 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:29:46,175::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:29:46,179::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:29:46,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:46,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913716 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:29:48,033::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:29:48,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:49,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919408 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:29:50,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:29:50,641::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:29:50,652::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 15:29:50,655::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656097 s, 797 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:29:50,656::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000610376 s, 608 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:51,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:51,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912161 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:53,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:53,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912285 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:55,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:55,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913871 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:29:56,186::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:29:56,188::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:29:57,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:29:57,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914901 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:29:59,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:00,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557967 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:30:00,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:30:00,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:30:00,690::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628187 s, 591 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:30:00,691::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000717331 s, 729 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:02,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:02,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914167 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:30:03,036::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd97d246-9449-41a9-81da-36807f023808`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:30:03,037::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:30:03,041::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:30:03,041::task::1191::Storage.TaskManager.Task::(prepare) Task=`dd97d246-9449-41a9-81da-36807f023808`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:30:03,041::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd97d246-9449-41a9-81da-36807f023808`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:30:03,041::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:30:03,041::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:30:03,042::task::993::Storage.TaskManager.Task::(_decref) Task=`dd97d246-9449-41a9-81da-36807f023808`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:30:03,068::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:30:04,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:04,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913575 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:30:06,195::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:30:06,196::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:30:06,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:06,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544804 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:08,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:09,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550944 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:30:10,696::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:30:10,707::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:30:10,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649469 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:30:10,723::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680161 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:11,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:11,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915966 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:13,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:13,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091159 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:15,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:30:16,212::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:30:16,217::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:30:16,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.564536 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:30:18,080::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:30:18,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:18,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913045 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:20,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:20,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913166 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:30:20,696::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:30:20,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:30:20,739::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:30:20,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000788462 s, 471 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:30:20,753::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648103 s, 807 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:22,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:22,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921269 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:24,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:24,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916014 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:30:26,222::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:30:26,223::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:30:26,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:26,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913018 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:29,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:29,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916223 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:30:30,756::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:30:30,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:30:30,778::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665923 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:30:30,780::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000729996 s, 716 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:31,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:31,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925813 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:30:33,093::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:30:33,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:33,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912913 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:35,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:35,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913052 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:30:36,236::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:30:36,237::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:30:37,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:37,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918626 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:39,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:39,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913772 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:30:40,782::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:30:40,794::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:30:40,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00608017 s, 61.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:30:40,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000618346 s, 846 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:41,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:41,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091338 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:43,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:44,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544469 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:30:46,247::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:30:46,251::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:30:46,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:46,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909799 s, 11.3 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:30:48,104::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:30:48,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:49,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091435 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:30:50,736::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:30:50,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:30:50,823::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:30:50,833::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654311 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:30:50,836::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000605363 s, 864 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:51,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:51,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913052 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:53,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:53,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913017 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:55,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:55,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910416 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:30:56,258::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:30:56,259::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:30:57,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:57,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914821 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:30:59,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:30:59,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913042 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:31:00,837::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:31:00,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:31:00,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627438 s, 591 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:31:00,865::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000765022 s, 684 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:01,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:01,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912487 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:31:03,109::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ec0f10e-c17f-4ba8-8acb-db88b71ebe7e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:31:03,110::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:31:03,112::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:31:03,113::task::1191::Storage.TaskManager.Task::(prepare) Task=`7ec0f10e-c17f-4ba8-8acb-db88b71ebe7e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:31:03,113::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ec0f10e-c17f-4ba8-8acb-db88b71ebe7e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:31:03,113::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:31:03,113::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:31:03,113::task::993::Storage.TaskManager.Task::(_decref) Task=`7ec0f10e-c17f-4ba8-8acb-db88b71ebe7e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:31:03,138::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:31:03,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:03,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915565 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:05,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:06,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913431 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:31:06,271::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:31:06,272::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:31:08,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:08,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912263 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:10,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:10,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914187 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:31:10,869::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:31:10,881::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:31:10,896::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00056587 s, 656 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:31:10,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691126 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:12,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:12,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914215 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:14,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:14,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913449 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:31:16,283::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:31:16,287::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:31:16,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:16,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914265 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:31:18,150::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:31:18,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:18,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914251 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:20,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:31:20,767::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:31:20,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914605 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:31:20,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:31:20,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:31:20,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699484 s, 530 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:31:20,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659357 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:22,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:22,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913059 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:24,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:25,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913171 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:31:26,293::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:31:26,295::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:31:27,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:27,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913691 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:29,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:29,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912483 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:31:30,929::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:31:30,940::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659752 s, 562 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:31:30,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:31:30,954::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069654 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:31,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:31,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914797 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:31:33,162::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:31:33,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:33,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923692 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:35,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:35,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913735 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:31:36,305::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:31:36,307::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:31:37,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:37,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914205 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:39,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:39,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091438 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:31:40,954::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:31:40,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:31:40,982::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000746462 s, 497 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:31:40,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650416 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:41,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:42,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913184 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:44,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:44,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915455 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:46,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:46,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100575 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:31:46,322::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:31:46,327::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:31:48,176::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:31:48,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:48,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913474 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:50,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:31:50,800::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 15:31:51,092::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:31:51,104::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:31:51,117::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645697 s, 810 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:31:51,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000806393 s, 460 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:51,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.59416 s, 1.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:53,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:53,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914646 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:55,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:55,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916557 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:31:56,327::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:31:56,329::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:31:57,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:31:57,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555481 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:31:59,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:00,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55366 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:32:01,128::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:32:01,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:32:01,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682762 s, 543 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:32:01,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000742792 s, 704 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:02,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:02,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921332 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:32:03,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a3b6cef-3bdf-4d9a-96cc-da248f156c87`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:32:03,179::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:32:03,184::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:32:03,184::task::1191::Storage.TaskManager.Task::(prepare) Task=`1a3b6cef-3bdf-4d9a-96cc-da248f156c87`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:32:03,184::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a3b6cef-3bdf-4d9a-96cc-da248f156c87`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:32:03,184::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:32:03,184::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:32:03,184::task::993::Storage.TaskManager.Task::(_decref) Task=`1a3b6cef-3bdf-4d9a-96cc-da248f156c87`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:32:03,212::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:32:04,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:05,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545685 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:32:06,339::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:32:06,340::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:32:07,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:07,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541581 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:09,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:09,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914646 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:32:11,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:32:11,168::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:32:11,179::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00421092 s, 88.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:32:11,180::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000627757 s, 833 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:11,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:12,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913185 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:14,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:14,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913357 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:16,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:32:16,352::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:32:16,357::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:32:16,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554987 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:32:18,224::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:32:18,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:18,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914491 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:32:20,834::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:32:20,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:20,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913863 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:32:21,185::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:32:21,196::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:32:21,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655442 s, 566 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:32:21,208::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660857 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:22,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:23,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912041 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:25,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:25,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913248 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:32:26,363::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:32:26,364::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:32:27,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:27,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914763 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:29,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:29,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913999 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:32:31,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:32:31,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671436 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:32:31,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:32:31,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636918 s, 821 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:31,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:31,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916865 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:32:33,248::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:32:33,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:33,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924675 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:35,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:35,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914232 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:32:36,373::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:32:36,374::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:32:37,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:37,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912984 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:39,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:40,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502408 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:32:41,243::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:32:41,253::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:32:41,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639108 s, 580 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:32:41,275::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:32:41,276::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:32:41,279::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:32:41,280::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:32:41,281::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:32:41,281::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:32:41,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:32:41,293::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684708 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:42,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:43,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550322 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:45,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:45,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550817 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:32:46,387::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:32:46,390::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:32:47,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:48,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541779 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:32:48,266::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:32:50,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:50,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.532148 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:32:50,867::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:32:51,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:32:51,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622262 s, 596 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:32:51,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:32:51,308::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000618211 s, 846 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:52,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:52,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913182 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:54,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:55,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542413 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:32:56,393::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:32:56,394::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:32:57,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:57,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.149966 s, 6.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:32:59,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:32:59,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912927 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:33:01,285::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:33:01,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066306 s, 560 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:33:01,312::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:33:01,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697538 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:01,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:02,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557842 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:33:03,269::task::595::Storage.TaskManager.Task::(_updateState) Task=`d84d4aa7-1be5-4599-b53a-35305adcdf87`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:33:03,269::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:33:03,272::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:33:03,272::task::1191::Storage.TaskManager.Task::(prepare) Task=`d84d4aa7-1be5-4599-b53a-35305adcdf87`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:33:03,272::task::595::Storage.TaskManager.Task::(_updateState) Task=`d84d4aa7-1be5-4599-b53a-35305adcdf87`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:33:03,273::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:33:03,273::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:33:03,273::task::993::Storage.TaskManager.Task::(_decref) Task=`d84d4aa7-1be5-4599-b53a-35305adcdf87`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:33:03,298::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:33:04,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:04,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912331 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:33:06,401::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:33:06,402::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:33:06,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:06,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913725 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:08,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:08,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915066 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:10,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:10,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913915 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:33:11,299::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:33:11,312::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:33:11,314::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:33:11,317::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:33:11,318::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:33:11,319::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:33:11,319::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:33:11,320::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:33:11,332::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636169 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:33:11,334::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:33:11,347::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000611495 s, 855 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:12,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:12,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913506 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:14,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:15,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915163 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:33:16,417::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:33:16,420::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:33:17,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:17,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913655 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:33:18,310::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:33:19,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:19,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916087 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:33:20,902::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:33:21,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:33:21,414::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:33:21,414::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:33:21,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00092653 s, 564 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:33:21,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068349 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:21,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.127773 s, 8.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:23,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:23,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913702 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:25,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:26,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561511 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:33:26,447::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:33:26,448::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:33:28,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:28,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551807 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:30,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:30,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915626 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:33:31,447::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:33:31,460::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629122 s, 590 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:33:31,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:33:31,474::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000715454 s, 731 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:32,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:32,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914143 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:33:33,322::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:33:34,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:35,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555207 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:33:36,467::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:33:36,469::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:33:37,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:38,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541705 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:40,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:40,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551135 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:33:41,475::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:33:41,488::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000614184 s, 604 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:33:41,491::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:33:41,504::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000617847 s, 846 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:42,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:42,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916327 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:44,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:44,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914306 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:33:46,482::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:33:46,488::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:33:46,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:47,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132419 s, 7.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:33:48,333::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:33:49,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:49,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912696 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:33:50,935::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:33:51,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:51,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914809 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:33:51,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:33:51,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:33:51,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066096 s, 561 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:33:51,531::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646488 s, 809 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:53,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:53,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913907 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:55,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:55,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912503 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:33:56,489::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:33:56,490::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:33:57,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:57,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915205 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:33:59,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:33:59,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.13305 s, 7.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:34:01,532::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:34:01,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:34:01,559::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650229 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:34:01,560::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642573 s, 814 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:01,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:01,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914049 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:34:03,336::task::595::Storage.TaskManager.Task::(_updateState) Task=`668a8e0b-b446-4656-86e5-074629e953e2`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:34:03,336::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:34:03,339::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:34:03,339::task::1191::Storage.TaskManager.Task::(prepare) Task=`668a8e0b-b446-4656-86e5-074629e953e2`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:34:03,339::task::595::Storage.TaskManager.Task::(_updateState) Task=`668a8e0b-b446-4656-86e5-074629e953e2`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:34:03,339::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:34:03,339::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:34:03,339::task::993::Storage.TaskManager.Task::(_decref) Task=`668a8e0b-b446-4656-86e5-074629e953e2`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:34:03,364::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:34:03,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:04,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920889 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:06,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:06,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913551 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:34:06,502::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:34:06,503::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:34:08,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:08,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:10,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:10,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915586 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:34:11,563::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:34:11,575::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:34:11,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677607 s, 548 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:34:11,589::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00105327 s, 497 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:12,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:12,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914445 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:14,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:14,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911641 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:34:16,515::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:34:16,520::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:34:16,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:16,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912568 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:34:18,376::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:34:18,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:18,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915312 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:20,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:34:20,968::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:34:21,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914307 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:34:21,591::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:34:21,603::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:34:21,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000694187 s, 534 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:34:21,617::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639585 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:23,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:23,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515177 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:25,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:25,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914792 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:34:26,521::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:34:26,522::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:34:27,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:27,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915334 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:29,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:29,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914974 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:34:31,620::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:34:31,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692897 s, 535 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:34:31,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:34:31,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000798213 s, 655 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:31,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:32,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913697 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:34:33,387::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:34:34,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:34,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913232 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:36,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:36,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914704 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:34:36,534::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:34:36,535::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:34:38,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:38,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915118 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:40,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:40,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914769 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:34:41,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:34:41,660::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650082 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:34:41,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:34:41,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697722 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:42,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:42,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914144 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:44,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:45,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555067 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:34:46,549::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:34:46,553::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:34:47,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:47,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132079 s, 7.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:34:48,398::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:34:49,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:49,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559241 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:34:51,002::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:34:51,666::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:34:51,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000610484 s, 608 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:34:51,681::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:34:51,693::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626296 s, 835 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:51,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:52,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913506 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:54,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:54,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912782 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:34:56,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:56,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913214 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:34:56,557::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:34:56,558::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:34:58,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:34:58,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915039 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:00,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:00,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915232 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:35:01,683::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:35:01,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620888 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:35:01,697::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:35:01,708::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665017 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:02,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:02,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913695 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:35:03,401::task::595::Storage.TaskManager.Task::(_updateState) Task=`1453121d-abd1-4b6c-80aa-62219a4d4fda`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:35:03,401::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:35:03,404::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:35:03,404::task::1191::Storage.TaskManager.Task::(prepare) Task=`1453121d-abd1-4b6c-80aa-62219a4d4fda`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:35:03,404::task::595::Storage.TaskManager.Task::(_updateState) Task=`1453121d-abd1-4b6c-80aa-62219a4d4fda`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:35:03,404::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:35:03,404::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:35:03,405::task::993::Storage.TaskManager.Task::(_decref) Task=`1453121d-abd1-4b6c-80aa-62219a4d4fda`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:35:03,430::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:35:04,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:04,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918383 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:35:06,569::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:35:06,570::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:35:06,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:06,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911821 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:08,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:09,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550447 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:11,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:11,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913438 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:35:11,698::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:35:11,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618217 s, 600 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:35:11,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:35:11,724::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650831 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:13,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:13,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925399 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:15,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:15,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0943042 s, 10.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:35:16,577::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:35:16,578::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:35:17,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:18,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913387 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:35:18,442::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:35:20,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:20,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914257 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:35:21,035::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:35:21,713::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:35:21,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000715782 s, 518 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:35:21,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:35:21,739::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675697 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:22,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:22,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545933 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:24,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:24,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911645 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:35:26,589::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:35:26,591::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:35:26,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:26,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913864 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:28,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:29,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914107 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:31,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:31,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915618 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:35:31,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:35:31,739::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000666257 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:35:31,742::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:35:31,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683698 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:33,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:33,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915619 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:35:33,453::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:35:35,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:35,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091233 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:35:36,597::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:35:36,598::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:35:37,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:37,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913633 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:39,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:39,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915695 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:41,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:35:41,796::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:35:41,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:35:41,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684825 s, 764 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:35:41,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000750131 s, 495 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:41,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.126356 s, 8.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:43,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:43,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913735 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:45,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:46,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913245 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:35:46,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:35:46,610::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:35:48,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:48,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914725 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:35:48,470::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:35:50,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:50,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913352 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:35:51,066::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 15:35:51,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:35:51,842::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:35:51,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00197976 s, 264 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:35:51,857::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670323 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:52,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:52,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913899 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:54,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:54,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913147 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:35:56,619::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:35:56,620::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:35:56,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:56,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912769 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:35:58,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:35:58,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913345 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:00,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:01,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916304 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:36:01,863::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:36:01,874::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:36:01,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000722711 s, 724 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:36:01,892::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000714335 s, 519 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:03,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:03,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914656 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:36:03,473::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4f4bce6-026c-4a73-a993-0ebb580f9a0f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:36:03,473::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:36:03,476::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:36:03,476::task::1191::Storage.TaskManager.Task::(prepare) Task=`c4f4bce6-026c-4a73-a993-0ebb580f9a0f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:36:03,477::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4f4bce6-026c-4a73-a993-0ebb580f9a0f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:36:03,477::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:36:03,477::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:36:03,477::task::993::Storage.TaskManager.Task::(_decref) Task=`c4f4bce6-026c-4a73-a993-0ebb580f9a0f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:36:03,503::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:36:05,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:05,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915367 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:36:06,631::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:36:06,633::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:36:07,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:07,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915917 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:09,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:09,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915467 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:11,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:11,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913526 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:36:11,899::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:36:11,899::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:36:11,924::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680124 s, 769 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:36:11,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705093 s, 526 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:13,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:14,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549706 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:16,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:16,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999699 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:36:16,644::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:36:16,645::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:36:18,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:36:18,524::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:36:18,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913485 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:20,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:20,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914391 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:36:21,100::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:36:21,933::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:36:21,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:36:21,955::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637447 s, 582 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:36:21,957::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645092 s, 811 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:22,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:23,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545415 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:25,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:25,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913087 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:36:26,651::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:36:26,652::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:36:27,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:27,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913889 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:29,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:29,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913946 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:31,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:31,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915131 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:36:31,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:36:31,973::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:36:31,984::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00428614 s, 86.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:36:31,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000751965 s, 696 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:36:33,536::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:36:33,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:33,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911684 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:35,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:35,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918554 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:36:36,664::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:36:36,665::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:36:37,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:38,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913981 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:40,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:40,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091488 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:36:41,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:36:42,000::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:36:42,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00538086 s, 68.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:36:42,012::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666341 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:42,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:42,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913292 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:44,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:44,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915753 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:46,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:46,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911994 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:36:46,676::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:36:46,677::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 15:36:48,547::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:36:48,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:48,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914538 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:50,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:50,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913731 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:36:51,133::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:36:52,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:36:52,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:36:52,040::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669624 s, 554 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:36:52,041::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652287 s, 802 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:52,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:52,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910356 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:54,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:55,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913775 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:36:56,688::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:36:56,689::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:36:57,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:57,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091028 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:36:59,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:36:59,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912741 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:01,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:01,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915087 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:37:02,047::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:37:02,058::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:37:02,073::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065777 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:37:02,076::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684587 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:03,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:37:03,549::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5a0ff20-2d9d-43bb-8cf9-4d146fc71f69`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:37:03,550::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:37:03,987::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:37:03,987::task::1191::Storage.TaskManager.Task::(prepare) Task=`f5a0ff20-2d9d-43bb-8cf9-4d146fc71f69`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:37:03,988::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5a0ff20-2d9d-43bb-8cf9-4d146fc71f69`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:37:03,988::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:37:03,989::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:37:03,990::task::993::Storage.TaskManager.Task::(_decref) Task=`f5a0ff20-2d9d-43bb-8cf9-4d146fc71f69`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 15:37:03,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556606 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:37:04,020::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:37:06,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:06,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917379 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:37:06,700::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:37:06,701::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:37:08,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:08,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914076 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:10,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:10,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915617 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:37:12,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:37:12,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000740853 s, 501 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:37:12,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:37:12,103::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000736684 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:12,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:12,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913073 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:14,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:14,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913878 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:16,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:16,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913423 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:37:16,716::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:37:16,720::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:37:18,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:18,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913784 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:37:19,032::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:37:20,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:20,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091427 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:37:21,166::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:37:22,104::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:37:22,116::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689608 s, 538 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:37:22,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:37:22,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00073525 s, 711 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:22,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:23,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912402 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:25,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:25,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557011 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:37:26,724::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:37:26,725::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:37:27,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:27,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914306 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:29,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:29,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913789 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:31,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:32,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913432 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:37:32,132::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:37:32,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:37:32,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00603785 s, 61.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:37:32,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663598 s, 788 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:37:34,044::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:37:34,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:34,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912968 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:36,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:36,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913747 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:37:36,736::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:37:36,738::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:37:38,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:39,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37601 s, 744 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:41,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:41,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914335 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:37:42,159::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:37:42,162::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:37:42,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00077322 s, 480 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:37:42,188::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:37:42,190::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:37:42,193::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:37:42,194::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:37:42,194::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:37:42,195::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:37:42,196::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:37:42,207::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000633342 s, 826 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:43,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:43,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914938 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:45,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:46,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0966802 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:37:46,745::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:37:46,746::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:37:48,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:48,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555457 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:37:49,056::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:37:50,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:50,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915599 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:37:51,196::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:37:52,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:37:52,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063721 s, 582 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:37:52,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:37:52,223::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679694 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:52,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:52,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914823 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:54,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:55,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914105 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:37:56,757::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:37:56,758::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:37:57,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:57,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914224 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:37:59,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:37:59,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912781 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:01,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:01,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910082 s, 11.3 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:38:02,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:38:02,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00062028 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:38:02,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:38:02,237::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693457 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:03,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:03,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913894 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:38:04,060::task::595::Storage.TaskManager.Task::(_updateState) Task=`ee9ee3f7-d408-4ff3-81d8-a01f95c1bc9e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:38:04,060::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:38:04,064::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:38:04,064::task::1191::Storage.TaskManager.Task::(prepare) Task=`ee9ee3f7-d408-4ff3-81d8-a01f95c1bc9e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:38:04,064::task::595::Storage.TaskManager.Task::(_updateState) Task=`ee9ee3f7-d408-4ff3-81d8-a01f95c1bc9e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:38:04,064::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:38:04,064::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:38:04,064::task::993::Storage.TaskManager.Task::(_decref) Task=`ee9ee3f7-d408-4ff3-81d8-a01f95c1bc9e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:38:04,092::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:38:05,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:05,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091468 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:38:06,776::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:38:06,777::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:38:07,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:07,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914209 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:09,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:10,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559629 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:38:12,213::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:38:12,228::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:38:12,229::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:38:12,232::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:38:12,234::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:38:12,234::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:38:12,234::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:38:12,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:38:12,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00071383 s, 520 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:38:12,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:38:12,261::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693165 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:12,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:12,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913665 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:14,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:15,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55966 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:38:16,788::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:38:16,789::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:38:17,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:17,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913389 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:38:19,104::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:38:19,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:19,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912811 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:38:21,230::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:38:21,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:21,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0953335 s, 10.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:38:22,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:38:22,275::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658642 s, 563 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:38:22,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:38:22,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000798583 s, 655 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:23,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:23,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09131 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:25,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:25,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912944 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:38:26,801::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:38:26,802::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:38:27,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:27,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913759 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:29,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:29,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914229 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:31,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:38:32,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:38:32,462::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:38:32,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00303227 s, 122 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:38:32,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000792077 s, 660 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:32,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561154 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:38:34,117::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:38:34,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:34,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909594 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:36,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:38:36,813::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:38:36,814::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:38:37,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.533665 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:39,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:39,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.534303 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:41,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:41,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912566 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:38:42,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:38:42,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:38:42,514::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000809091 s, 459 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:38:42,516::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00108259 s, 483 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:43,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:44,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913443 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:46,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:46,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101507 s, 10.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:38:46,821::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:38:46,822::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:38:48,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:48,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915393 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:38:49,136::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:38:50,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:50,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912534 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:38:51,262::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:38:52,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:38:52,530::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:38:52,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00397477 s, 93.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:38:52,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000821281 s, 637 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:52,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:52,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912802 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:54,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:54,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913482 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:56,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:38:56,834::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:38:56,835::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:38:56,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913493 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:38:58,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:38:59,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911061 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:01,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:01,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914099 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:39:02,548::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:39:02,561::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000419984 s, 883 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:39:02,563::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:39:02,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000605456 s, 864 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:03,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:03,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091522 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:39:04,139::task::595::Storage.TaskManager.Task::(_updateState) Task=`16b0e4b9-08cc-4bcc-8047-fcacb9aa8dff`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:39:04,139::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:39:04,142::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:39:04,142::task::1191::Storage.TaskManager.Task::(prepare) Task=`16b0e4b9-08cc-4bcc-8047-fcacb9aa8dff`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:39:04,142::task::595::Storage.TaskManager.Task::(_updateState) Task=`16b0e4b9-08cc-4bcc-8047-fcacb9aa8dff`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:39:04,142::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:39:04,143::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:39:04,143::task::993::Storage.TaskManager.Task::(_decref) Task=`16b0e4b9-08cc-4bcc-8047-fcacb9aa8dff`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:39:04,170::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:39:05,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:05,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091344 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:39:06,839::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:39:06,840::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:39:07,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:07,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914536 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:09,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:10,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544402 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:12,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:12,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912799 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:39:12,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:39:12,589::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:39:12,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000771963 s, 481 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:39:12,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621955 s, 841 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:14,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:14,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:16,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:16,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0996135 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:39:16,848::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:39:16,849::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:39:18,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:19,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556739 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:39:19,183::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:39:21,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:21,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914238 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:39:21,293::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:39:22,607::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:39:22,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:39:22,629::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620473 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:39:22,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006786 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:23,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:23,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913916 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:25,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:25,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914507 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:39:26,860::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:39:26,861::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:39:27,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:27,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913263 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:29,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:29,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913608 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:31,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:31,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916476 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:39:32,635::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:39:32,655::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:39:32,656::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662226 s, 560 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:39:32,669::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668053 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:33,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:33,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914525 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:39:34,194::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:39:35,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:36,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915478 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:39:36,872::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:39:36,873::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:39:38,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:38,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918644 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:40,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:40,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915496 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:42,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:42,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913902 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:39:42,673::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:39:42,686::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682452 s, 544 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:39:42,686::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:39:42,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000714243 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:44,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:44,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914738 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:46,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:39:46,884::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:39:46,885::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:39:47,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541743 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:49,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:39:49,208::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:39:49,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922497 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:51,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:39:51,325::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:39:51,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550394 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:39:52,700::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:39:52,711::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:39:52,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000737016 s, 503 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:39:52,726::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000729394 s, 717 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:53,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:54,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551431 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:56,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:39:56,894::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:39:56,895::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:39:56,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543157 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:39:58,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:39:59,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914498 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:01,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:01,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915272 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:40:02,727::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:40:02,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:40:02,749::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068681 s, 540 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:40:02,750::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000727072 s, 719 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:03,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:03,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091548 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:40:04,211::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fd8603e-3f00-48f5-bb21-541853e48f79`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:40:04,212::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:40:04,215::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:40:04,215::task::1191::Storage.TaskManager.Task::(prepare) Task=`1fd8603e-3f00-48f5-bb21-541853e48f79`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:40:04,215::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fd8603e-3f00-48f5-bb21-541853e48f79`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:40:04,215::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:40:04,215::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:40:04,215::task::993::Storage.TaskManager.Task::(_decref) Task=`1fd8603e-3f00-48f5-bb21-541853e48f79`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:40:04,243::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:40:05,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:05,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914271 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:40:06,906::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:40:06,907::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:40:07,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:07,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914673 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:09,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:10,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559549 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:12,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:12,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917325 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:40:12,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:40:12,765::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:40:12,777::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000631935 s, 587 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:40:12,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000634599 s, 824 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:14,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:14,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914467 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:16,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:40:16,918::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:40:16,919::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:40:16,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542429 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:18,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:19,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091326 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:40:19,255::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:40:21,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:21,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913789 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:40:21,357::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 15:40:22,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:40:22,796::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:40:22,807::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00430969 s, 86.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:40:22,808::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069777 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:23,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:23,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914815 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:25,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:25,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912343 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:40:26,930::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:40:26,931::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:40:27,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:27,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914187 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:29,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:30,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541741 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:32,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:32,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914007 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:40:32,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:40:32,827::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:40:32,842::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667343 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:40:32,844::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621228 s, 842 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:40:34,268::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:40:34,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:34,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912879 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:36,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:40:36,942::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:40:36,944::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:40:36,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913694 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:38,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:39,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912806 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:41,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:41,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556874 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:40:42,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:40:42,859::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:40:42,870::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000980822 s, 378 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:40:42,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641489 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:43,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:43,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913018 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:45,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:45,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968355 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:40:46,954::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:40:46,955::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:40:48,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:48,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915387 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:40:49,281::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:40:50,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:50,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914193 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:40:51,385::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:40:52,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:52,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913682 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:40:52,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:40:52,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:40:52,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678555 s, 547 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:40:52,905::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631766 s, 828 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:54,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:54,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913099 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:40:56,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:56,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091513 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:40:56,966::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:40:56,968::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:40:58,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:40:58,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927891 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:00,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:00,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915241 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:02,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:41:02,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:41:02,969::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:41:02,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663568 s, 559 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:41:02,985::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641613 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:02,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129736 s, 7.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:41:04,285::task::595::Storage.TaskManager.Task::(_updateState) Task=`b431a532-97bf-4d0c-b801-d705601bc636`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:41:04,285::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:41:04,288::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:41:04,288::task::1191::Storage.TaskManager.Task::(prepare) Task=`b431a532-97bf-4d0c-b801-d705601bc636`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:41:04,288::task::595::Storage.TaskManager.Task::(_updateState) Task=`b431a532-97bf-4d0c-b801-d705601bc636`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:41:04,288::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:41:04,288::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:41:04,288::task::993::Storage.TaskManager.Task::(_decref) Task=`b431a532-97bf-4d0c-b801-d705601bc636`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:41:04,315::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:41:05,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:05,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551305 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:41:06,979::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:41:06,980::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:41:07,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:07,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912453 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:09,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:10,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546546 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:12,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:12,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914264 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:41:12,992::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:41:13,004::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:41:13,015::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000827936 s, 448 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:41:13,018::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688996 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:14,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:14,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091355 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:16,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:16,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913423 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:41:16,991::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:41:16,992::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:41:18,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:18,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912758 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:41:19,327::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:41:20,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:20,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913237 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:41:21,425::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:41:22,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:41:23,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:41:23,406::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:41:23,421::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000695351 s, 534 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:41:23,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00156653 s, 334 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:23,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552216 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:25,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:25,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09139 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:41:27,003::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:41:27,004::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:41:27,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:27,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914121 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:29,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:29,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914054 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:31,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:31,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913523 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:41:33,429::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:41:33,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000796766 s, 466 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:41:33,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:41:33,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000811569 s, 644 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:33,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:34,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913502 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:41:34,339::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:41:36,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:36,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915026 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:41:37,015::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:41:37,016::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:41:38,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:38,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913699 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:40,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:40,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915151 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:42,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:42,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091755 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:41:43,456::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:41:43,467::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:41:43,478::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0060038 s, 61.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:41:43,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000722363 s, 724 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:44,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:45,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39605 s, 734 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:41:47,027::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:41:47,028::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:41:48,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:48,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543127 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:41:49,351::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:41:50,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:50,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914154 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:41:51,459::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:41:52,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:52,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914575 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:41:53,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:41:53,486::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:41:53,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000634193 s, 585 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:41:53,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000915769 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:41:54,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:56,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36302 s, 751 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:41:57,040::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:41:57,041::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:41:58,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:41:58,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914255 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:00,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:00,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.537732 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:02,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:03,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545969 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:42:03,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:42:03,528::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:42:03,539::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00479044 s, 77.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:42:03,541::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694206 s, 753 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:42:04,354::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f0b1edd-b9f2-4558-9cc1-386822b03302`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:42:04,355::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:42:04,358::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:42:04,358::task::1191::Storage.TaskManager.Task::(prepare) Task=`0f0b1edd-b9f2-4558-9cc1-386822b03302`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:42:04,358::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f0b1edd-b9f2-4558-9cc1-386822b03302`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:42:04,358::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:42:04,358::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:42:04,359::task::993::Storage.TaskManager.Task::(_decref) Task=`0f0b1edd-b9f2-4558-9cc1-386822b03302`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:42:04,387::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:42:05,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:05,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912671 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:42:07,052::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:42:07,053::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:42:07,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:07,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913316 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:09,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:09,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913448 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:11,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:11,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915419 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:42:13,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:42:13,555::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627718 s, 591 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:42:13,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:42:13,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000633439 s, 826 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:13,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:14,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913579 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:16,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:16,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10075 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:42:17,064::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:42:17,065::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:42:18,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:18,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914341 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:42:19,400::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:42:20,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:20,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914972 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:42:21,491::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:42:22,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:22,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913378 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:42:23,570::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:42:23,582::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:42:23,582::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000804149 s, 461 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:42:23,597::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720438 s, 726 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:24,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:24,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912852 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:26,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:26,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913504 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:42:27,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:42:27,078::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:42:28,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:29,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915027 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:31,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:31,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916052 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:33,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:33,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916235 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:42:33,598::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:42:33,611::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00078808 s, 471 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:42:33,611::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:42:33,625::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000793277 s, 659 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:42:34,412::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:42:35,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:35,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913505 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:42:37,086::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:42:37,087::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:42:37,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:37,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915521 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:39,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:40,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557982 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:42,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:42,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092156 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:42:43,625::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:42:43,635::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:42:43,639::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00136352 s, 272 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:42:43,657::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:42:43,658::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:42:43,661::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:42:43,662::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:42:43,663::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:42:43,663::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:42:43,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:42:43,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637736 s, 820 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:44,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:44,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913532 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:46,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:46,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100364 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:42:47,098::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:42:47,099::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:42:48,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:49,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559364 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:42:49,426::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:42:51,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:51,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914163 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:42:51,528::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:42:53,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:42:53,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:42:53,777::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:42:53,791::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000613351 s, 605 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:42:53,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000709689 s, 737 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:53,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.534508 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:55,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:42:57,110::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:42:57,111::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:42:57,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38188 s, 741 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:42:59,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:42:59,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0936182 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:01,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:01,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914587 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:03,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:43:04,032::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:43:04,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:43:04,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641998 s, 578 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:43:04,064::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000784167 s, 667 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:04,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.598995 s, 1.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:43:04,429::task::595::Storage.TaskManager.Task::(_updateState) Task=`6cb7c8fe-08d6-41ae-984d-9fd57c86026c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:43:04,429::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:43:04,432::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:43:04,432::task::1191::Storage.TaskManager.Task::(prepare) Task=`6cb7c8fe-08d6-41ae-984d-9fd57c86026c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:43:04,432::task::595::Storage.TaskManager.Task::(_updateState) Task=`6cb7c8fe-08d6-41ae-984d-9fd57c86026c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:43:04,433::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:43:04,433::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:43:04,433::task::993::Storage.TaskManager.Task::(_decref) Task=`6cb7c8fe-08d6-41ae-984d-9fd57c86026c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:43:04,458::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:43:06,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:06,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0960621 s, 10.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:43:07,122::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:43:07,123::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:43:08,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:08,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916115 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:10,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:10,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916864 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:12,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:13,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.564705 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:43:14,069::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 15:43:14,081::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:43:14,096::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636563 s, 822 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:43:14,108::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:43:14,109::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:43:14,112::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:43:14,114::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:43:14,114::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:43:14,114::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:43:14,115::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:43:14,127::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657571 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:15,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:15,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914184 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:43:17,131::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:43:17,132::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:43:17,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:17,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913689 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:19,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:19,471::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921526 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:43:19,475::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:43:21,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:43:21,560::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:43:21,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915014 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:23,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:23,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914833 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:43:24,105::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:43:24,117::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000702152 s, 745 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:43:24,130::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:43:24,141::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668852 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:25,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:25,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913083 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:43:27,140::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:43:27,141::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:43:27,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:27,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913655 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:29,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:30,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09134 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:32,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:32,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552343 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:43:34,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:43:34,133::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068115 s, 768 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:43:34,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:43:34,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691854 s, 536 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:43:34,488::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:43:34,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:34,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912994 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:36,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:43:37,153::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:43:37,154::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:43:37,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560079 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:39,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:39,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0941203 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:41,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:41,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913922 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:43,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:43,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914934 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:43:44,137::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:43:44,149::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668025 s, 783 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:43:44,158::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:43:44,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000694994 s, 534 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:45,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:46,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557849 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:43:47,160::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:43:47,161::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:43:48,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:48,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913962 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:43:49,500::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:43:50,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:50,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914623 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:43:51,592::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:43:52,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:52,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914934 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:43:54,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:43:54,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662978 s, 789 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:43:54,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:43:54,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063206 s, 587 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:54,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:55,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55092 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:43:57,170::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:43:57,171::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:43:57,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:43:57,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548781 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:43:59,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:00,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546349 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:02,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:02,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915016 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:44:04,169::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:44:04,181::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000712597 s, 734 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:44:04,188::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:44:04,200::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664588 s, 558 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:44:04,503::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ff21094-19be-48e0-8f98-9814e8ee4017`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:44:04,504::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:44:04,506::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:44:04,506::task::1191::Storage.TaskManager.Task::(prepare) Task=`7ff21094-19be-48e0-8f98-9814e8ee4017`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:44:04,507::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ff21094-19be-48e0-8f98-9814e8ee4017`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:44:04,507::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:44:04,507::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:44:04,507::task::993::Storage.TaskManager.Task::(_decref) Task=`7ff21094-19be-48e0-8f98-9814e8ee4017`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:44:04,532::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:44:04,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:04,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913168 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:06,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:06,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915201 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:44:07,179::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:44:07,180::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:44:08,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:08,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916531 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:10,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:11,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914455 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:13,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:13,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918026 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:44:14,185::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:44:14,197::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068586 s, 763 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:44:14,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:44:14,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652368 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:15,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:15,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914806 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:44:17,188::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:44:17,189::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:44:17,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:17,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914924 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:19,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:44:19,546::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:44:19,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914865 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:21,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:44:21,624::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:44:21,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914512 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:23,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:23,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091952 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:44:24,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:44:24,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681989 s, 767 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:44:24,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:44:24,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636154 s, 583 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:25,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:25,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927491 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:44:27,200::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:44:27,201::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:44:27,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:28,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09139 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:30,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:30,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091433 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:32,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:32,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556907 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:44:34,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:44:34,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00199898 s, 262 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:44:34,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:44:34,244::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661607 s, 561 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:44:34,558::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:44:34,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:34,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912281 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:36,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:37,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914364 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:44:37,217::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:44:37,218::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:44:39,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:39,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917942 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:41,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:41,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091466 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:43,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:43,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915589 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:44:44,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:44:44,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661053 s, 791 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:44:44,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:44:44,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689884 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:45,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:45,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913209 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:44:47,229::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:44:47,230::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:44:47,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:48,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555328 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:44:49,570::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:44:50,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:50,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552117 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:44:51,657::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:44:52,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:52,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913212 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:44:54,265::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:44:54,277::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:44:54,287::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000729174 s, 717 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:44:54,289::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0010451 s, 355 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:54,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:54,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913013 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:44:56,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:56,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914676 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:44:57,236::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:44:57,238::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:44:59,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:44:59,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915309 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:01,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:01,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915242 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:03,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:03,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913351 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:45:04,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:45:04,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:45:04,320::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000711372 s, 522 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:45:04,321::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062613 s, 835 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:45:04,573::task::595::Storage.TaskManager.Task::(_updateState) Task=`90dd4d6a-bd4b-4f96-959e-c675d6f85f08`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:45:04,573::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:45:04,576::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:45:04,576::task::1191::Storage.TaskManager.Task::(prepare) Task=`90dd4d6a-bd4b-4f96-959e-c675d6f85f08`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:45:04,576::task::595::Storage.TaskManager.Task::(_updateState) Task=`90dd4d6a-bd4b-4f96-959e-c675d6f85f08`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:45:04,576::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:45:04,577::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:45:04,577::task::993::Storage.TaskManager.Task::(_decref) Task=`90dd4d6a-bd4b-4f96-959e-c675d6f85f08`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:45:04,603::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:45:05,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:05,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915267 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:45:07,248::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:45:07,249::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:45:07,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:07,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09144 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:09,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:09,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913364 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:11,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:12,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54196 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:14,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:45:14,371::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:45:14,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620627 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:45:14,387::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:45:14,403::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000781413 s, 669 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:14,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129198 s, 7.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:16,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:16,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101425 s, 10.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:45:17,260::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:45:17,262::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:45:18,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:19,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541819 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:45:19,616::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:45:21,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:21,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091417 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:45:21,689::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:45:23,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:23,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911243 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:45:24,403::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:45:24,415::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:45:24,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652557 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:45:24,427::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652846 s, 801 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:25,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:25,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557353 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:45:27,273::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:45:27,274::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:45:27,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:28,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091501 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:30,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:30,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544475 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:32,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:32,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914806 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:45:34,432::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:45:34,443::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:45:34,444::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000813849 s, 643 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:45:34,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000748116 s, 496 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:45:34,627::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:45:34,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:35,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542544 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:45:37,280::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:45:37,281::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:45:37,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:37,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542446 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:39,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:40,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091419 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:42,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:42,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913499 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:44,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:44,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915096 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:45:44,465::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:45:44,466::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:45:44,487::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00368904 s, 101 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:45:44,488::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000728937 s, 717 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:46,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:46,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914514 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:45:47,292::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:45:47,294::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:45:48,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:48,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091901 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:45:49,639::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:45:50,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:50,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913258 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:45:51,721::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:45:52,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:52,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913158 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:54,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:45:54,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:45:54,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:45:55,004::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668031 s, 783 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:45:55,005::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623776 s, 595 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:55,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0979222 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:45:57,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:57,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916655 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:45:57,305::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:45:57,306::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:45:59,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:45:59,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915012 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:01,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:01,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916149 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:03,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:03,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913828 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:46:04,642::task::595::Storage.TaskManager.Task::(_updateState) Task=`2fffd405-aa0e-4f97-9cdb-c4756c119e84`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:46:04,642::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:46:04,645::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:46:04,646::task::1191::Storage.TaskManager.Task::(prepare) Task=`2fffd405-aa0e-4f97-9cdb-c4756c119e84`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:46:04,646::task::595::Storage.TaskManager.Task::(_updateState) Task=`2fffd405-aa0e-4f97-9cdb-c4756c119e84`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:46:04,646::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:46:04,646::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:46:04,646::task::993::Storage.TaskManager.Task::(_decref) Task=`2fffd405-aa0e-4f97-9cdb-c4756c119e84`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:46:04,673::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:46:05,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:46:05,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:46:05,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617089 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:46:05,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000779509 s, 671 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:05,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:05,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914461 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:46:07,317::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:46:07,318::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:46:07,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:07,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913824 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:09,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:09,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091436 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:11,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:11,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914131 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:13,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:14,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914684 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:46:15,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:46:15,054::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:46:15,069::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00105803 s, 351 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:46:15,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00079264 s, 660 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:16,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:16,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912623 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:46:17,329::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:46:17,330::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:46:18,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:18,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091407 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:46:19,685::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:46:20,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:20,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922204 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:46:21,754::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:46:22,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:22,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915013 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:24,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:46:25,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:46:25,193::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:46:25,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621736 s, 597 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:46:25,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000909748 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:25,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.577107 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:27,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:27,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09133 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:46:27,345::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:46:27,352::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:46:29,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:29,471::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915473 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:31,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:31,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914246 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:33,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:33,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921356 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:46:34,697::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:46:35,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:46:35,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:46:35,223::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000722281 s, 514 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:46:35,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687683 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:35,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:36,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556492 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:46:37,351::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:46:37,352::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:46:38,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:38,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914054 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:40,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:40,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557712 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:43,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:43,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916573 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:45,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:46:45,657::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:46:45,658::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:46:45,684::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000704915 s, 742 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:46:45,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00071838 s, 516 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:45,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.565607 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:46:47,363::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:46:47,364::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:46:47,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:47,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914562 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:46:49,709::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:46:49,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:50,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.115378 s, 8.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:46:51,787::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:46:52,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:52,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092268 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:54,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:54,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915177 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:46:55,698::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:46:55,710::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:46:55,721::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0058501 s, 63.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:46:55,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00075888 s, 689 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:46:56,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:56,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913861 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:46:57,376::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:46:57,377::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:46:58,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:46:58,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917166 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:00,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:00,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915043 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:02,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:02,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914732 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:47:04,712::task::595::Storage.TaskManager.Task::(_updateState) Task=`8945275f-7ed6-4d33-9b8d-947a57c87a4f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:47:04,712::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:47:04,715::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:47:04,715::task::1191::Storage.TaskManager.Task::(prepare) Task=`8945275f-7ed6-4d33-9b8d-947a57c87a4f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:47:04,716::task::595::Storage.TaskManager.Task::(_updateState) Task=`8945275f-7ed6-4d33-9b8d-947a57c87a4f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:47:04,716::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:47:04,716::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:47:04,716::task::993::Storage.TaskManager.Task::(_decref) Task=`8945275f-7ed6-4d33-9b8d-947a57c87a4f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:47:04,743::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:47:04,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:05,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545599 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:47:05,726::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:47:05,737::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:47:05,748::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680242 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:47:05,751::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698744 s, 748 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:07,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:47:07,400::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:47:07,401::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:47:07,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915366 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:09,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:09,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915017 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:11,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:12,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546474 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:14,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:14,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914041 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:47:15,755::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:47:15,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:47:15,780::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000810754 s, 458 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:47:15,785::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000751132 s, 696 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:16,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:16,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911698 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:47:17,399::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:47:17,400::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:47:18,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:19,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561234 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:47:19,754::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:47:21,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:21,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541653 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:47:21,820::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:47:23,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:23,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913089 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:47:25,789::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:47:25,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000759954 s, 488 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:47:25,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:47:25,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066637 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:25,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:25,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912749 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:47:27,407::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:47:27,408::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:47:27,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:28,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914326 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:30,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:30,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556035 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:32,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:32,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914443 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:34,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:47:34,776::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:47:34,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917345 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:47:35,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:47:35,827::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:47:35,839::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00550246 s, 67.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:47:35,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00072448 s, 722 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:36,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:47:37,419::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:47:37,420::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:47:37,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555649 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:39,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:39,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916387 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:41,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:41,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915402 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:43,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:44,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.553341 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:47:45,843::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:47:45,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:47:45,859::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000923592 s, 402 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:47:45,903::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:47:45,905::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:47:45,910::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:47:45,914::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:47:45,915::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:47:45,916::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:47:45,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:47:45,933::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639297 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:46,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:46,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091558 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:47:47,429::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:47:47,430::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:47:48,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:48,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914439 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:47:49,788::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:47:50,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:50,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913443 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:47:51,864::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:47:52,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:52,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916161 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:47:54,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:55,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544343 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:47:55,870::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:47:55,882::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063327 s, 586 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:47:55,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:47:55,952::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621616 s, 841 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:47:57,442::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:47:57,443::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:47:57,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:47:58,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544426 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:00,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:00,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0947199 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:02,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:02,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915007 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:04,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:48:04,791::task::595::Storage.TaskManager.Task::(_updateState) Task=`cf5de2a5-91f7-4ae9-a529-6a91e6f74cd5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:48:04,791::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:48:04,798::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:48:04,798::task::1191::Storage.TaskManager.Task::(prepare) Task=`cf5de2a5-91f7-4ae9-a529-6a91e6f74cd5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:48:04,798::task::595::Storage.TaskManager.Task::(_updateState) Task=`cf5de2a5-91f7-4ae9-a529-6a91e6f74cd5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:48:04,799::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:48:04,799::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:48:04,799::task::993::Storage.TaskManager.Task::(_decref) Task=`cf5de2a5-91f7-4ae9-a529-6a91e6f74cd5`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 15:48:04,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.527289 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:48:04,831::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:48:05,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:48:05,897::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670732 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:48:05,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:48:05,968::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677487 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:06,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:06,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912669 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:48:07,453::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:48:07,454::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:48:08,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:09,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914981 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:11,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:11,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55911 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:13,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:13,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914073 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:15,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:15,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916109 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:48:15,899::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:48:15,952::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:48:15,955::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:48:15,960::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:48:15,965::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:48:15,966::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:48:15,967::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:48:15,974::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:48:15,989::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:48:16,003::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622346 s, 596 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:48:16,008::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000722985 s, 723 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:48:17,465::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:48:17,466::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:48:18,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:18,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914707 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:48:19,845::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:48:20,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:20,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912819 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:48:21,894::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:48:22,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:22,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914756 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:24,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:24,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913379 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:48:26,016::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:48:26,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:48:26,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668774 s, 555 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:48:26,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688232 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:26,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:27,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550893 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:48:27,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:48:27,475::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:48:29,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:29,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913616 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:31,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:31,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551062 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:33,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:33,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913498 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:48:34,858::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:48:35,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:35,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914847 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:48:36,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:48:36,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617544 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:48:36,057::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:48:36,069::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067912 s, 770 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:48:37,486::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:48:37,487::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:48:38,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:38,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916102 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:40,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:40,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914041 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:42,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:42,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:44,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:44,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101814 s, 10.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:48:46,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:48:46,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:48:46,160::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062124 s, 842 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:48:46,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000763154 s, 486 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:46,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:46,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910317 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:48:47,498::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:48:47,499::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:48:48,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:48,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912447 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:48:49,870::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:48:50,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:51,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559805 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:48:51,925::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:48:53,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:53,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:55,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:55,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914212 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:48:56,173::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:48:56,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:48:56,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665794 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:48:56,196::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659045 s, 794 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:48:57,507::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:48:57,508::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:48:57,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:57,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913531 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:48:59,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:48:59,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091374 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:01,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:01,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913096 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:03,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:04,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091329 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:49:04,873::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ee98285-5db6-4fb0-9948-99215bc5c691`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:49:04,874::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:49:04,877::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:49:04,877::task::1191::Storage.TaskManager.Task::(prepare) Task=`4ee98285-5db6-4fb0-9948-99215bc5c691`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:49:04,877::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ee98285-5db6-4fb0-9948-99215bc5c691`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:49:04,877::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:49:04,877::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:49:04,878::task::993::Storage.TaskManager.Task::(_decref) Task=`4ee98285-5db6-4fb0-9948-99215bc5c691`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:49:04,905::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:49:06,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:06,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918806 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:49:06,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:49:06,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:49:06,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686711 s, 540 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:49:06,228::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703505 s, 743 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:49:07,519::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:49:07,520::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:49:08,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:08,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913362 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:10,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:10,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557439 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:12,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:12,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912622 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:14,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:15,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.533064 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:49:16,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:49:16,234::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:49:16,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663499 s, 559 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:49:16,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000834673 s, 627 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:17,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:49:17,532::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:49:17,534::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:49:17,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913873 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:19,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:19,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914131 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:49:19,917::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:49:21,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:21,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913533 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:49:21,958::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:49:23,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:23,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913256 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:25,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:26,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913194 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:49:26,266::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:49:26,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:49:26,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665108 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:49:26,291::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660461 s, 792 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:49:27,544::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:49:27,545::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:49:28,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:28,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558806 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:30,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:30,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913575 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:32,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:32,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915177 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:34,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:49:34,939::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:49:35,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915276 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:49:36,292::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:49:36,304::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:49:36,319::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0007759 s, 478 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:49:36,320::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684984 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:37,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:37,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915642 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:49:37,556::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:49:37,557::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:49:39,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:39,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091318 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:41,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:41,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914124 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:43,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:43,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542799 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:45,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:46,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0983461 s, 10.4 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:49:46,326::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:49:46,339::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:49:46,342::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000607279 s, 611 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:49:46,357::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000749488 s, 698 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:49:47,565::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:49:47,566::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:49:48,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:48,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0953944 s, 10.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:49:49,951::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:49:50,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:50,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914961 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:49:51,991::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:49:52,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:53,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555544 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:55,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:55,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921779 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:49:56,363::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:49:56,374::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:49:56,390::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069848 s, 531 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:49:56,391::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00075149 s, 696 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:57,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:49:57,575::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:49:57,576::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:49:57,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554543 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:49:59,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:49:59,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918313 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:01,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:01,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913727 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:03,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:04,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543278 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:50:04,954::task::595::Storage.TaskManager.Task::(_updateState) Task=`6fc2acfb-66e9-4c49-8d4e-963c8e595def`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:50:04,955::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:50:04,957::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:50:04,957::task::1191::Storage.TaskManager.Task::(prepare) Task=`6fc2acfb-66e9-4c49-8d4e-963c8e595def`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:50:04,957::task::595::Storage.TaskManager.Task::(_updateState) Task=`6fc2acfb-66e9-4c49-8d4e-963c8e595def`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:50:04,958::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:50:04,958::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:50:04,958::task::993::Storage.TaskManager.Task::(_decref) Task=`6fc2acfb-66e9-4c49-8d4e-963c8e595def`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:50:04,985::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 15:50:06,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:50:06,406::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:50:06,422::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660551 s, 562 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:50:06,424::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000745055 s, 702 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:06,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:06,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091423 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:50:07,588::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:50:07,589::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:50:08,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:08,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913622 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:10,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:10,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914817 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:12,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:12,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919793 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:15,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:15,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916164 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:50:16,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:50:16,431::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:50:16,452::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00534319 s, 69.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:50:16,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687048 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:17,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:17,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912279 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:50:17,599::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:50:17,600::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:50:19,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:19,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915373 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:50:19,998::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:50:21,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:21,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910258 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:50:22,024::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:50:23,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:23,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911424 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:25,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:25,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913817 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:50:26,459::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:50:26,471::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000708657 s, 524 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:50:26,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:50:26,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687718 s, 760 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:50:27,612::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:50:27,613::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:50:27,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:27,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914587 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:29,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:30,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911489 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:32,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:32,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914238 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:34,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:34,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913028 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:50:35,010::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:50:36,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:36,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091353 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:50:36,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:50:36,497::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652534 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:50:36,499::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:50:36,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716189 s, 730 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:50:37,620::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:50:37,621::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:50:38,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:38,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.51212 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:40,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:41,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550219 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:43,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:43,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913732 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:45,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:46,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557741 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:50:46,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:50:46,533::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683866 s, 543 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:50:46,533::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:50:46,551::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642617 s, 814 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:50:47,630::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:50:47,631::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:50:48,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:48,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913311 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:50:50,030::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:50:50,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:51,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558702 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:50:52,055::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:50:53,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:53,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913398 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:55,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:55,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912932 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:50:56,552::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:50:56,564::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:50:56,565::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000745866 s, 497 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:50:56,579::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00213148 s, 245 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:50:57,642::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:50:57,643::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:50:57,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:50:57,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911894 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:50:59,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:00,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913748 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:02,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:02,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913759 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:04,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:04,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913082 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:51:05,034::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1c6de1d-0b9f-485e-ae6f-c572e86f6288`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:51:05,036::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:51:05,094::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:51:05,094::task::1191::Storage.TaskManager.Task::(prepare) Task=`a1c6de1d-0b9f-485e-ae6f-c572e86f6288`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:51:05,094::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1c6de1d-0b9f-485e-ae6f-c572e86f6288`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:51:05,094::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:51:05,095::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:51:05,095::task::993::Storage.TaskManager.Task::(_decref) Task=`a1c6de1d-0b9f-485e-ae6f-c572e86f6288`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:51:05,122::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:51:06,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:06,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915858 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:51:06,581::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:51:06,592::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:51:06,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00432192 s, 85.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:51:06,605::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683986 s, 765 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:51:07,655::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:51:07,656::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:51:08,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:08,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091339 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:10,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:10,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917032 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:12,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:12,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913744 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:14,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:15,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546205 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:51:16,610::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:51:16,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:51:16,623::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618197 s, 600 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:51:16,636::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00102453 s, 510 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:17,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:17,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912637 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:51:17,667::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:51:17,668::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:51:19,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:19,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914147 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:51:20,135::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:51:21,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:21,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919088 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:51:22,085::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:51:23,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:23,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913316 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:25,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:51:26,660::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:51:26,671::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:51:26,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000745106 s, 702 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:51:26,690::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00062047 s, 598 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:26,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554448 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:51:27,679::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:51:27,680::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:51:28,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:28,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910683 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:30,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:30,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914569 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:32,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:33,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557836 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:51:35,151::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:51:35,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:35,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915675 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:51:36,702::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:51:36,714::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:51:36,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00072279 s, 513 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:51:36,727::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690384 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:37,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:51:37,693::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:51:37,694::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:51:37,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913342 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:39,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:40,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547924 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:42,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:42,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924826 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:44,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:45,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560026 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:47,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:51:47,204::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:51:47,215::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:51:47,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000724381 s, 722 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:51:47,299::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00129696 s, 286 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:47,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106116 s, 9.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:51:47,701::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:51:47,702::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:51:49,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:49,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914987 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:51:50,165::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:51:51,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:51,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912712 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:51:52,117::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:51:53,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:53,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914513 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:55,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:55,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914053 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:51:57,307::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:51:57,319::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:51:57,330::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639354 s, 580 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:51:57,332::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000802144 s, 652 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:51:57,713::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:51:57,715::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:51:57,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:51:57,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914497 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:51:59,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:00,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914488 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:02,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:02,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926764 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:04,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:04,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916716 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:52:05,167::task::595::Storage.TaskManager.Task::(_updateState) Task=`6bd76c03-16aa-4959-b883-ec3df00adea5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:52:05,168::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:52:05,171::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:52:05,171::task::1191::Storage.TaskManager.Task::(prepare) Task=`6bd76c03-16aa-4959-b883-ec3df00adea5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:52:05,171::task::595::Storage.TaskManager.Task::(_updateState) Task=`6bd76c03-16aa-4959-b883-ec3df00adea5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:52:05,172::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:52:05,172::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:52:05,172::task::993::Storage.TaskManager.Task::(_decref) Task=`6bd76c03-16aa-4959-b883-ec3df00adea5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:52:05,198::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:52:06,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:06,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914582 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:52:07,335::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:52:07,346::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:52:07,361::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677099 s, 548 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:52:07,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000714359 s, 732 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:52:07,725::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:52:07,726::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:52:08,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:08,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913985 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:10,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:10,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914329 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:12,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:12,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914368 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:14,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:14,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915677 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:16,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:16,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913043 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:52:17,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:52:17,377::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:52:17,387::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660699 s, 562 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:52:17,390::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652843 s, 801 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:52:17,737::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:52:17,739::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:52:18,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:19,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913696 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:52:20,209::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:52:21,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:21,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09146 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:52:22,151::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:52:23,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:23,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913691 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:25,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:25,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914373 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:52:27,394::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:52:27,405::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:52:27,416::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000721875 s, 514 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:52:27,417::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069266 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:27,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:27,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913463 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:52:27,750::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:52:27,751::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:52:29,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:29,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913241 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:31,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:31,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913172 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:33,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:33,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912552 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:52:35,227::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:52:35,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:36,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552157 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:52:37,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:52:37,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000813136 s, 456 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:52:37,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:52:37,446::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663676 s, 788 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:52:37,761::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:52:37,762::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:52:38,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:38,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916411 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:40,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:40,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914414 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:42,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:43,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544618 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:45,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:45,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559209 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:52:47,447::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:52:47,457::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:52:47,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066598 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:52:47,478::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:52:47,479::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:52:47,482::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:52:47,484::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:52:47,484::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:52:47,484::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:52:47,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:52:47,497::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679568 s, 770 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:52:47,773::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:52:47,774::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:52:47,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:48,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914653 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:50,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:50,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091302 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:52:50,239::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 15:52:52,184::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:52:52,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:52,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913327 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:54,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:54,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091268 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:52:56,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:56,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914556 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:52:57,471::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:52:57,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00064989 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:52:57,500::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:52:57,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006638 s, 788 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:52:57,781::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:52:57,782::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:52:58,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:52:59,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551518 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:01,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:01,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914327 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:03,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:03,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915016 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:53:05,242::task::595::Storage.TaskManager.Task::(_updateState) Task=`716b8445-de53-4254-a434-7fa59a21aead`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:53:05,243::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:53:05,245::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:53:05,246::task::1191::Storage.TaskManager.Task::(prepare) Task=`716b8445-de53-4254-a434-7fa59a21aead`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:53:05,246::task::595::Storage.TaskManager.Task::(_updateState) Task=`716b8445-de53-4254-a434-7fa59a21aead`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:53:05,246::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:53:05,246::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:53:05,246::task::993::Storage.TaskManager.Task::(_decref) Task=`716b8445-de53-4254-a434-7fa59a21aead`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:53:05,273::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:53:05,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:05,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913573 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:07,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:53:07,496::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:53:07,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681523 s, 544 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:53:07,589::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:53:07,605::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00073617 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:07,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.116689 s, 8.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:53:07,793::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:53:07,794::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:53:09,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:09,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912449 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:11,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:11,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914537 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:13,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:13,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919474 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:15,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:16,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0998511 s, 10.3 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:53:17,605::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 15:53:17,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:53:17,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000747223 s, 700 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:53:17,642::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:53:17,643::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:53:17,646::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:53:17,648::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:53:17,648::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:53:17,648::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:53:17,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:53:17,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628443 s, 590 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:53:17,805::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:53:17,806::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:53:18,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:18,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914323 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:53:20,286::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:53:20,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:20,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911994 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:53:22,218::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:53:22,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:22,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915287 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:24,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:25,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54975 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:27,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:27,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913427 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:53:27,640::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:53:27,652::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000727599 s, 719 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:53:27,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:53:27,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665523 s, 557 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:53:27,817::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:53:27,818::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:53:29,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:29,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913509 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:31,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:31,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913078 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:33,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:34,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.494918 s, 2.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:53:35,298::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:53:36,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:36,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091549 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:53:37,655::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:53:37,667::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686591 s, 762 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:53:37,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:53:37,690::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661024 s, 561 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:53:37,829::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:53:37,837::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:53:38,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:38,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913693 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:40,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:40,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914309 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:42,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:42,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913639 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:44,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:44,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:46,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:47,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550455 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:53:47,671::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:53:47,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621673 s, 841 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:53:47,693::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:53:47,704::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661093 s, 561 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:53:47,837::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:53:47,838::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:53:49,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:49,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914003 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:53:50,311::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:53:51,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:51,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915678 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:53:52,252::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:53:53,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:53,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914247 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:55,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:55,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915854 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:57,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:53:57,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:53:57,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:53:57,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00101557 s, 515 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:53:57,801::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660453 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:53:57,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.115829 s, 8.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:53:57,847::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:53:57,848::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:53:59,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:53:59,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913866 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:01,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:02,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555803 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:04,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:04,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914826 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:54:05,314::task::595::Storage.TaskManager.Task::(_updateState) Task=`9dca2c99-dd46-4bb9-9644-090a55378599`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:54:05,314::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:54:05,317::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:54:05,317::task::1191::Storage.TaskManager.Task::(prepare) Task=`9dca2c99-dd46-4bb9-9644-090a55378599`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:54:05,317::task::595::Storage.TaskManager.Task::(_updateState) Task=`9dca2c99-dd46-4bb9-9644-090a55378599`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:54:05,317::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:54:05,317::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:54:05,318::task::993::Storage.TaskManager.Task::(_decref) Task=`9dca2c99-dd46-4bb9-9644-090a55378599`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:54:05,344::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:54:06,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:07,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542685 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:54:07,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:54:07,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:54:07,838::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622701 s, 596 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:54:07,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660191 s, 792 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:54:07,856::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:54:07,857::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:54:09,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:09,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542844 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:11,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:11,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911821 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:13,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:14,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914609 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:16,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:16,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0987595 s, 10.4 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:54:17,845::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:54:17,856::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:54:17,870::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00552631 s, 67.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:54:17,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000718971 s, 727 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:54:17,886::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:54:17,887::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:54:18,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:18,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.255737 s, 4.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:54:20,356::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:54:20,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:20,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915142 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:54:22,284::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:54:22,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:22,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913043 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:24,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:25,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913695 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:27,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:27,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912935 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:54:27,883::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:54:27,895::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:54:27,911::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672116 s, 778 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:54:27,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000835227 s, 444 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:54:27,922::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:54:27,925::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:54:29,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:29,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091421 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:31,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:31,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914272 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:33,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:33,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913985 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:54:35,369::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:54:35,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:35,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091447 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:37,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:37,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913101 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:54:37,923::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:54:37,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675247 s, 549 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:54:37,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:54:37,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655999 s, 797 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:54:37,957::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:54:37,967::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:54:39,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:39,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920085 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:41,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:41,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915274 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:43,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:44,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913951 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:46,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:47,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39667 s, 733 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:54:47,955::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:54:47,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:54:47,968::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:54:47,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00222778 s, 167 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:54:47,973::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:54:47,985::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000614936 s, 850 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:49,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:50,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.537596 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:54:50,381::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:54:52,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:52,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914642 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:54:52,314::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:54:54,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:54,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910353 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:56,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:56,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09186 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:54:57,966::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:54:57,967::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:54:57,975::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:54:57,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628022 s, 591 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:54:57,990::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:54:58,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000622858 s, 840 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:54:58,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:54:58,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915249 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:00,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:00,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0929117 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:02,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:02,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:04,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:05,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551706 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:55:05,384::task::595::Storage.TaskManager.Task::(_updateState) Task=`d664d682-9fb1-44fc-b713-8ace1176451d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:55:05,384::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:55:05,387::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:55:05,388::task::1191::Storage.TaskManager.Task::(prepare) Task=`d664d682-9fb1-44fc-b713-8ace1176451d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:55:05,388::task::595::Storage.TaskManager.Task::(_updateState) Task=`d664d682-9fb1-44fc-b713-8ace1176451d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:55:05,388::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:55:05,388::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:55:05,388::task::993::Storage.TaskManager.Task::(_decref) Task=`d664d682-9fb1-44fc-b713-8ace1176451d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:55:05,415::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:55:07,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:07,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091381 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:55:07,977::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:55:07,978::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:55:07,990::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:55:08,002::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630128 s, 589 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:55:08,005::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:55:08,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698188 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:09,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:09,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914225 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:11,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:11,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915169 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:13,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:13,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912405 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:15,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:15,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921738 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:55:17,990::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:55:17,991::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:55:17,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 15:55:18,007::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:55:18,097::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069617 s, 533 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:55:18,099::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:55:18,116::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000730168 s, 716 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:18,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.123507 s, 8.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:20,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:20,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914198 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:55:20,427::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:55:22,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:55:22,348::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:55:22,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558869 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:24,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:24,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914006 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:26,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:55:27,998::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:55:27,999::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 15:55:28,336::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:55:28,347::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:55:28,358::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00538175 s, 97.2 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:55:28,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617008 s, 601 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:28,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39563 s, 734 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:30,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:30,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548414 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:32,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:33,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916297 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:35,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:35,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914283 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:55:35,439::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:55:37,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:37,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914671 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:55:38,010::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:55:38,011::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:55:38,367::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:55:38,379::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697373 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:55:38,381::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:55:38,394::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684085 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:39,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:39,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0933732 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:41,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:41,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913636 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:43,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:44,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558566 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:46,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:46,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546547 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:55:48,021::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:55:48,022::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:55:48,394::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:55:48,406::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629603 s, 589 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:55:48,408::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:55:48,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067158 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:48,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:48,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912542 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:55:50,452::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:55:50,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:51,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.569256 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:55:52,392::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:55:53,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:53,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913033 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:55,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:55,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091914 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:57,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:57,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914578 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:55:58,031::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:55:58,033::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:55:58,421::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:55:58,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:55:58,448::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000767803 s, 483 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:55:58,450::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655189 s, 798 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:55:59,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:55:59,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914631 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:01,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:01,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0929538 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:04,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:04,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915961 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:56:05,455::task::595::Storage.TaskManager.Task::(_updateState) Task=`c694c37d-3c8a-4653-9948-6d7c010272bb`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:56:05,455::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:56:05,458::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:56:05,458::task::1191::Storage.TaskManager.Task::(prepare) Task=`c694c37d-3c8a-4653-9948-6d7c010272bb`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:56:05,458::task::595::Storage.TaskManager.Task::(_updateState) Task=`c694c37d-3c8a-4653-9948-6d7c010272bb`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:56:05,458::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:56:05,459::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:56:05,459::task::993::Storage.TaskManager.Task::(_decref) Task=`c694c37d-3c8a-4653-9948-6d7c010272bb`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:56:05,484::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:56:06,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:06,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913499 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:56:08,044::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:56:08,045::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:56:08,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:08,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917075 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:56:08,453::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:56:08,464::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:56:08,475::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659033 s, 563 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:56:08,476::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00111165 s, 470 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:10,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:10,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914528 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:12,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:13,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544556 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:15,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:15,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915388 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:17,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:17,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551869 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:56:18,056::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:56:18,057::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:56:18,481::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:56:18,493::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:56:18,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00078523 s, 472 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:56:18,507::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000752423 s, 695 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:19,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:19,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914522 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:56:20,495::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:56:21,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:21,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913447 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:56:22,422::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:56:23,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:24,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.578135 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:26,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:26,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914485 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:56:28,068::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:56:28,069::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:56:28,508::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:56:28,520::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000732745 s, 506 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:56:28,521::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:56:28,533::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700137 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:28,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:29,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559117 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:31,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:31,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914087 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:33,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:33,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09137 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:56:35,507::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:56:35,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:36,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544207 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:56:38,077::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:56:38,078::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:56:38,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:38,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913751 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:56:38,534::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:56:38,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:56:38,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000789786 s, 470 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:56:38,563::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680182 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:40,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:40,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914532 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:42,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:42,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913005 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:44,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:44,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913831 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:46,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:46,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914517 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:56:48,086::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:56:48,087::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:56:48,568::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:56:48,580::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:56:48,580::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688729 s, 539 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:56:48,593::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641769 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:48,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:48,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914221 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:56:50,519::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:56:50,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:50,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091263 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:56:52,462::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:56:52,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:53,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546325 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:55,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:56:56,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55032 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:56:58,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:56:58,099::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:56:58,100::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:56:58,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913045 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:56:58,595::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:56:58,608::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000608245 s, 610 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:56:58,609::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:56:58,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655294 s, 798 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:00,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:00,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916589 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:02,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:02,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092409 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:04,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:04,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920487 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:57:05,522::task::595::Storage.TaskManager.Task::(_updateState) Task=`d0ba0010-2bbe-4acd-9dfd-86af752595de`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:57:05,522::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:57:05,525::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:57:05,525::task::1191::Storage.TaskManager.Task::(prepare) Task=`d0ba0010-2bbe-4acd-9dfd-86af752595de`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:57:05,525::task::595::Storage.TaskManager.Task::(_updateState) Task=`d0ba0010-2bbe-4acd-9dfd-86af752595de`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:57:05,525::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:57:05,526::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:57:05,526::task::993::Storage.TaskManager.Task::(_decref) Task=`d0ba0010-2bbe-4acd-9dfd-86af752595de`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:57:05,554::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:57:06,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:06,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912285 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:57:08,111::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:57:08,112::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:57:08,621::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:57:08,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:57:08,644::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650765 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:57:08,645::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691662 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:08,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:08,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912764 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:10,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:10,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914527 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:12,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:12,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912509 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:15,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:15,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914045 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:17,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:17,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910851 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:57:18,120::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:57:18,121::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:57:18,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:57:18,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:57:18,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680488 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:57:18,674::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716546 s, 730 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:19,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:19,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561616 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:57:20,566::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:57:21,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:21,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914411 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:57:22,495::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:57:23,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:24,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913679 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:26,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:26,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543079 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:57:28,132::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:57:28,133::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:57:28,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 15:57:28,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:57:28,749::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:57:28,761::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707689 s, 739 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:57:28,765::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00149403 s, 248 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:28,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.125244 s, 8.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:30,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:30,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912315 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:32,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:33,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915406 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:35,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:35,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914762 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:57:35,579::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:57:37,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:37,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914204 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:57:38,145::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:57:38,146::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:57:38,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:57:38,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:57:38,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681509 s, 544 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:57:38,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0012111 s, 432 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:39,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:39,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912321 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:41,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:41,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913958 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:43,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:43,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091528 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:45,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:45,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0980052 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:47,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:47,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920767 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:57:48,157::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:57:48,158::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:57:48,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:57:48,806::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 15:57:48,820::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000700278 s, 530 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:57:48,834::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 15:57:48,836::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 15:57:48,838::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 15:57:48,840::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 15:57:48,840::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 15:57:48,840::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 15:57:48,841::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:57:48,853::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676546 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:49,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:50,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914301 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:57:50,591::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:57:52,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 15:57:52,527::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:57:52,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.505789 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:54,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:54,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914026 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:56,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:56,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913861 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:57:58,169::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:57:58,170::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:57:58,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:57:58,839::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635959 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:57:58,856::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:57:58,869::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662088 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:57:58,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:57:58,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913621 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:00,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:01,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912949 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:03,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:03,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911223 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:05,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:58:05,594::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f33a8c6-2023-4445-ac6e-5fee4ea8b137`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:58:05,594::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:58:05,799::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:58:05,800::task::1191::Storage.TaskManager.Task::(prepare) Task=`4f33a8c6-2023-4445-ac6e-5fee4ea8b137`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:58:05,801::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f33a8c6-2023-4445-ac6e-5fee4ea8b137`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:58:05,802::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:58:05,802::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Dummy-49::DEBUG::2014-07-14 15:58:05,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.565047 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:58:05,804::task::993::Storage.TaskManager.Task::(_decref) Task=`4f33a8c6-2023-4445-ac6e-5fee4ea8b137`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:58:05,835::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:58:07,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:07,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091359 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:58:08,190::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:58:08,191::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:58:08,842::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:58:08,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655345 s, 566 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:58:08,883::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:58:08,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685665 s, 763 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:09,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:10,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914683 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:12,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:12,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914713 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:14,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:14,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914501 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:16,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:16,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0992222 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:58:18,202::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:58:18,203::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:58:18,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:18,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914629 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:58:18,856::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:58:18,870::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 15:58:18,871::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 15:58:18,874::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 15:58:18,876::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 15:58:18,876::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 15:58:18,876::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 15:58:18,877::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:58:18,889::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654251 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:58:18,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:58:18,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066076 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:20,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:20,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913759 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:58:20,848::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 15:58:22,560::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:58:22,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:22,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914902 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:24,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:24,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914526 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:26,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:27,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917673 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:58:28,213::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:58:28,215::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:58:28,893::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:58:28,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000722044 s, 514 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:58:28,914::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:58:28,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000605901 s, 863 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:29,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:29,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913995 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:31,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:31,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914889 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:33,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:33,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912572 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:35,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 15:58:35,861::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:58:36,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555682 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:38,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:38,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913748 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:58:38,226::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:58:38,227::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:58:38,907::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:58:38,919::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000605173 s, 613 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:58:38,930::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:58:38,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669562 s, 781 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:40,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:40,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554588 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:42,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:42,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912888 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:44,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:44,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914076 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:46,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:47,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913998 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:58:48,233::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:58:48,235::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:58:48,923::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:58:48,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667526 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:58:48,944::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:58:48,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692052 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:49,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:49,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914642 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:58:50,873::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:58:51,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:51,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913333 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:58:52,593::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:58:53,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:53,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914363 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:55,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:55,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:57,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:58:57,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914563 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:58:58,245::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:58:58,246::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:58:58,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:58:58,950::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642154 s, 578 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:58:58,960::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:58:58,971::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649673 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:58:59,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:00,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558134 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:02,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:02,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913844 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:04,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:04,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913127 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:59:05,877::task::595::Storage.TaskManager.Task::(_updateState) Task=`d5afa981-2167-40be-8976-40e843146595`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 15:59:05,877::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 15:59:05,880::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:59:05,880::task::1191::Storage.TaskManager.Task::(prepare) Task=`d5afa981-2167-40be-8976-40e843146595`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 15:59:05,880::task::595::Storage.TaskManager.Task::(_updateState) Task=`d5afa981-2167-40be-8976-40e843146595`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 15:59:05,880::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 15:59:05,881::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 15:59:05,881::task::993::Storage.TaskManager.Task::(_decref) Task=`d5afa981-2167-40be-8976-40e843146595`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 15:59:05,907::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:59:06,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:06,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913854 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:59:08,253::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:59:08,254::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:59:08,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:08,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0977084 s, 10.5 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:59:08,954::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:59:08,965::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628256 s, 591 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:59:08,974::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:59:08,986::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662699 s, 789 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:10,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:10,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915092 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:12,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:12,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914507 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:15,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:15,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091462 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:17,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:17,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912863 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:59:18,265::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:59:18,266::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:59:18,969::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:59:18,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000930519 s, 399 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:59:18,989::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:59:19,002::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684455 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:19,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:19,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556537 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:59:20,919::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:59:21,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:21,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914181 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:59:22,625::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:59:23,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:24,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913838 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:26,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:26,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091339 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:28,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:59:28,278::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:59:28,279::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:59:28,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0950965 s, 10.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:59:28,985::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:59:28,996::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685436 s, 541 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:59:29,006::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:59:29,018::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000610954 s, 856 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:30,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:30,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913728 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:32,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:32,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555939 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:34,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:35,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916898 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:59:35,930::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:59:37,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:37,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914647 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:59:38,290::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:59:38,291::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 15:59:38,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:59:39,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648446 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:59:39,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:59:39,040::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000725551 s, 721 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:39,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:39,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913791 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:41,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:41,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913987 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:43,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:43,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913203 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:45,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:46,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542979 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:48,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 15:59:48,303::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:59:48,304::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:59:48,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917617 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:59:49,015::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:59:49,027::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693606 s, 535 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:59:49,052::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:59:49,064::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000730335 s, 716 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:50,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:50,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926199 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 15:59:50,944::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 15:59:52,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:52,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913822 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 15:59:52,658::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 15:59:54,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:54,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911632 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 15:59:56,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:56,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914728 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 15:59:58,309::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 15:59:58,310::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 15:59:58,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 15:59:58,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913182 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 15:59:59,031::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 15:59:59,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626726 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 15:59:59,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 15:59:59,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639748 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:00,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:01,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913833 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:03,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:03,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925376 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:05,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:05,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924537 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:00:05,947::task::595::Storage.TaskManager.Task::(_updateState) Task=`941e88e0-0a4d-4b96-aead-fad38762e43d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:00:05,947::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:00:05,950::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:00:05,950::task::1191::Storage.TaskManager.Task::(prepare) Task=`941e88e0-0a4d-4b96-aead-fad38762e43d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:00:05,950::task::595::Storage.TaskManager.Task::(_updateState) Task=`941e88e0-0a4d-4b96-aead-fad38762e43d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:00:05,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:00:05,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:00:05,951::task::993::Storage.TaskManager.Task::(_decref) Task=`941e88e0-0a4d-4b96-aead-fad38762e43d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:00:05,979::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:00:07,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:07,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915285 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:00:08,321::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:00:08,322::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:00:09,047::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:00:09,058::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006184 s, 600 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:00:09,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:00:09,095::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667857 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:09,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:09,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918792 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:11,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:11,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915313 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:13,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:13,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914319 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:15,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:15,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097997 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:18,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:18,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913958 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:00:18,333::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:00:18,334::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:00:19,062::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:00:19,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684539 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:00:19,099::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:00:19,111::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667054 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:20,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:20,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913375 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:00:20,991::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:00:22,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:22,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914244 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:00:22,695::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:00:24,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:24,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550821 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:26,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:27,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914325 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:00:28,339::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:00:28,341::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:00:29,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:00:29,089::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677051 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:29,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:00:29,119::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:00:29,209::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000728034 s, 718 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:29,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0969963 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:31,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:31,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091078 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:33,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:33,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913974 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:35,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:35,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091385 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:00:36,004::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:00:37,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:38,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543838 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:00:38,352::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:00:38,353::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:00:39,093::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:00:39,104::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661778 s, 561 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:00:39,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:00:39,228::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703689 s, 743 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:40,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:40,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542103 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:42,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:43,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542588 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:45,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:45,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915623 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:47,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:48,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.553782 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:00:48,364::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:00:48,365::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:00:49,108::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:00:49,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000711337 s, 522 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:00:49,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:00:49,243::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000710311 s, 736 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:50,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:50,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914878 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:00:51,019::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:00:52,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:52,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543434 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:00:52,727::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:00:54,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:54,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0941054 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:00:56,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:56,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913616 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:00:58,372::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:00:58,373::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:00:58,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:00:59,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914256 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:00:59,123::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:00:59,135::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636672 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:00:59,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:00:59,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713557 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:01,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:01,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914935 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:03,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:03,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0950577 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:05,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:05,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922758 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:01:06,022::task::595::Storage.TaskManager.Task::(_updateState) Task=`03cf6986-9bd1-4c97-a386-3bb1376db944`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:01:06,023::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:01:06,026::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:01:06,026::task::1191::Storage.TaskManager.Task::(prepare) Task=`03cf6986-9bd1-4c97-a386-3bb1376db944`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:01:06,026::task::595::Storage.TaskManager.Task::(_updateState) Task=`03cf6986-9bd1-4c97-a386-3bb1376db944`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:01:06,026::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:01:06,026::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:01:06,027::task::993::Storage.TaskManager.Task::(_decref) Task=`03cf6986-9bd1-4c97-a386-3bb1376db944`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:01:06,055::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:01:07,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:07,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558282 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:01:08,384::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:01:08,386::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:01:09,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:01:09,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620253 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:01:09,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:01:09,275::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678419 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:10,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:10,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924221 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:12,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:12,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920078 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:14,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:14,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542413 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:16,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:16,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913174 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:01:18,396::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:01:18,398::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:01:18,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:19,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913553 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:01:19,155::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:01:19,166::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692008 s, 536 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:01:19,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:01:19,291::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713255 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:21,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:01:21,068::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:01:21,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544546 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:01:22,759::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:01:23,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:23,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091503 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:25,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:25,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914942 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:27,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:27,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919529 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:01:28,406::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:01:28,407::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:01:29,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:01:29,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648163 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:01:29,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:01:29,307::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000721271 s, 725 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:29,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:30,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558661 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:32,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:32,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913414 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:34,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:34,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917834 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:01:36,080::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:01:36,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:37,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561814 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:01:38,418::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:01:38,419::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:01:39,185::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:01:39,197::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658552 s, 563 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:01:39,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:01:39,321::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683709 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:39,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:39,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545767 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:41,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:42,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104667 s, 9.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:44,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:44,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914729 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:46,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:46,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100486 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:01:48,427::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:01:48,428::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:01:48,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:48,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915427 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:01:49,200::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:01:49,212::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00072344 s, 513 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:01:49,325::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:01:49,336::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068395 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:50,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:50,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914678 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:01:51,101::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:01:52,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:01:52,791::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:01:52,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914776 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:54,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:54,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912796 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:01:56,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:57,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914039 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:01:58,437::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:01:58,438::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:01:59,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:01:59,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914531 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:01:59,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:01:59,228::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650644 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:01:59,340::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:01:59,352::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650389 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:01,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:01,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:03,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:03,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912959 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:05,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:05,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914554 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:02:06,104::task::595::Storage.TaskManager.Task::(_updateState) Task=`8875bba1-201e-43b5-8446-e4880caab67d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:02:06,104::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:02:06,108::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:02:06,108::task::1191::Storage.TaskManager.Task::(prepare) Task=`8875bba1-201e-43b5-8446-e4880caab67d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:02:06,108::task::595::Storage.TaskManager.Task::(_updateState) Task=`8875bba1-201e-43b5-8446-e4880caab67d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:02:06,108::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:02:06,109::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:02:06,109::task::993::Storage.TaskManager.Task::(_decref) Task=`8875bba1-201e-43b5-8446-e4880caab67d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:02:06,136::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:02:07,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:08,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552089 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:02:08,445::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:02:08,446::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:02:09,468::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:02:09,469::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:02:09,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00391884 s, 94.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:02:09,492::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00105454 s, 496 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:10,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:10,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916012 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:12,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:12,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913902 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:14,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:14,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913405 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:16,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:17,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544755 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:02:18,463::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:02:18,471::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:02:19,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:02:19,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:02:19,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:02:19,996::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692483 s, 755 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:02:19,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642049 s, 578 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:20,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.589419 s, 1.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:02:21,149::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:02:22,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:22,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915178 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:02:22,821::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:02:24,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:24,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913523 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:26,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:26,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913546 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:28,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:02:28,473::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:02:28,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:02:28,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0940758 s, 10.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:02:30,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:02:30,073::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:02:30,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000913432 s, 573 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:02:30,085::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000602316 s, 616 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:30,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:30,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0962531 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:32,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:32,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912777 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:34,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:34,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912847 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:02:36,162::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:02:36,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:37,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558877 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:02:38,485::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:02:38,487::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:02:39,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:39,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915361 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:02:40,094::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:02:40,094::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:02:40,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000714195 s, 519 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:02:40,121::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658419 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:41,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:41,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914441 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:43,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:43,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915519 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:45,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:45,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999715 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:48,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:48,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917556 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:02:48,493::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:02:48,494::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 16:02:50,124::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 16:02:50,125::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:02:50,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000995936 s, 373 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:02:50,157::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:02:50,159::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:02:50,162::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:02:50,163::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:02:50,163::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:02:50,163::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:02:50,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:02:50,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680967 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:50,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:50,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913433 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:02:51,185::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:02:52,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:52,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924708 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:02:52,853::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:02:54,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:54,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913643 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:02:56,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:02:57,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39677 s, 733 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:02:58,503::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:02:58,504::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:02:59,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:03:00,537::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:03:00,549::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:03:00,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000925919 s, 565 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:03:00,571::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665724 s, 557 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:00,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.584494 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:02,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:02,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091479 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:04,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:04,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914439 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:03:06,188::task::595::Storage.TaskManager.Task::(_updateState) Task=`2e9b0459-a05e-4639-be05-b5619fcbfab0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:03:06,189::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:03:06,192::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:03:06,192::task::1191::Storage.TaskManager.Task::(prepare) Task=`2e9b0459-a05e-4639-be05-b5619fcbfab0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:03:06,192::task::595::Storage.TaskManager.Task::(_updateState) Task=`2e9b0459-a05e-4639-be05-b5619fcbfab0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:03:06,192::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:03:06,192::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:03:06,193::task::993::Storage.TaskManager.Task::(_decref) Task=`2e9b0459-a05e-4639-be05-b5619fcbfab0`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:03:06,219::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:03:06,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:06,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915931 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:03:08,511::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:03:08,512::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:03:08,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:09,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915356 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:03:10,580::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:03:10,581::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:03:10,609::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639182 s, 818 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:03:10,609::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625002 s, 594 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:11,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:11,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915445 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:13,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:13,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914203 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:15,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:15,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915076 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:17,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:17,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091417 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:03:18,523::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:03:18,524::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:03:19,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:19,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914517 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:03:20,614::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 16:03:20,621::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:03:20,636::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664046 s, 788 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:03:20,649::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:03:20,650::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:03:20,653::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:03:20,654::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:03:20,655::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:03:20,655::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:03:20,656::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:03:20,667::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656161 s, 565 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:03:21,231::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:03:21,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:03:22,886::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:03:23,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39508 s, 734 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:25,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:25,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549554 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:27,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:27,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913703 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:03:28,534::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:03:28,535::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:03:29,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:30,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.532572 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:03:30,645::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:03:30,657::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000704612 s, 742 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:03:30,670::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:03:30,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000726134 s, 511 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:32,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:32,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913301 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:34,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:34,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546658 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:03:36,243::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:03:37,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:37,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0929575 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:03:38,546::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:03:38,547::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:03:39,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:39,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915099 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:03:40,660::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:03:40,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688263 s, 760 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:03:40,686::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:03:40,698::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664597 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:41,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:41,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922136 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:43,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:43,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913799 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:45,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:45,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100504 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:47,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:47,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914463 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:03:48,554::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:03:48,555::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:03:49,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:49,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915058 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:03:50,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:03:50,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687649 s, 761 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:03:50,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:03:50,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000702735 s, 528 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:03:51,255::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:03:51,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:52,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0936145 s, 10.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:03:52,920::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:03:54,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:54,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912573 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:56,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:03:56,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913771 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:03:58,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:03:58,563::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:03:58,564::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:03:58,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.567541 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:04:00,692::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:04:00,704::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673499 s, 777 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:04:00,715::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:04:00,727::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637084 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:01,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:01,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913478 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:03,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:03,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916279 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:05,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:05,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914295 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:04:06,258::task::595::Storage.TaskManager.Task::(_updateState) Task=`e5908a16-b458-48e6-832a-72d1e1ec6f4b`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:04:06,258::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:04:06,261::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:04:06,261::task::1191::Storage.TaskManager.Task::(prepare) Task=`e5908a16-b458-48e6-832a-72d1e1ec6f4b`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:04:06,262::task::595::Storage.TaskManager.Task::(_updateState) Task=`e5908a16-b458-48e6-832a-72d1e1ec6f4b`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:04:06,262::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:04:06,262::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:04:06,262::task::993::Storage.TaskManager.Task::(_decref) Task=`e5908a16-b458-48e6-832a-72d1e1ec6f4b`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:04:06,288::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:04:07,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:07,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915144 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:04:08,575::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:04:08,576::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:04:09,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:09,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091517 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:04:10,707::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:04:10,719::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645553 s, 810 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:04:10,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:04:10,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000610933 s, 607 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:11,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:11,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914191 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:13,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:13,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918433 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:15,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:16,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0993412 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:18,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:18,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919837 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:04:18,585::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:04:18,586::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:04:20,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:20,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915339 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:04:20,724::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:04:20,735::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666705 s, 784 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:04:20,745::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:04:20,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643604 s, 576 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:04:21,300::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:04:22,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:22,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913973 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:04:22,951::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:04:24,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:24,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091478 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:26,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:26,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913188 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:04:28,597::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:04:28,598::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:04:28,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:29,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915083 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:04:30,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:04:30,751::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678005 s, 771 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:04:30,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:04:30,771::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675355 s, 549 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:31,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:31,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559116 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:33,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:33,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091449 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:35,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:35,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914343 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:04:36,312::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:04:37,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:38,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913031 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:04:38,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:04:38,611::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:04:40,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:40,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915217 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:04:40,755::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:04:40,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000709616 s, 737 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:04:40,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:04:40,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674573 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:42,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:42,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914055 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:44,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:44,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913733 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:46,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:46,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.098552 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:48,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:04:48,620::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:04:48,621::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:04:48,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915169 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:50,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:04:51,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:04:51,260::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:04:51,271::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00476847 s, 77.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:04:51,276::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00166549 s, 314 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:51,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.592502 s, 1.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:04:51,324::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:04:52,984::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:04:53,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:53,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544368 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:55,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:55,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09136 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:04:57,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:04:58,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914755 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:04:58,632::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:04:58,634::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:05:00,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:00,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912114 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:05:01,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:05:01,291::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:05:01,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632557 s, 587 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:05:01,308::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000605143 s, 864 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:02,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:02,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915512 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:04,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:04,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914224 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:05:06,328::task::595::Storage.TaskManager.Task::(_updateState) Task=`eaa08b33-1a66-4682-9d5f-45ac67a5cef8`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:05:06,328::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:05:06,331::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:05:06,332::task::1191::Storage.TaskManager.Task::(prepare) Task=`eaa08b33-1a66-4682-9d5f-45ac67a5cef8`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:05:06,332::task::595::Storage.TaskManager.Task::(_updateState) Task=`eaa08b33-1a66-4682-9d5f-45ac67a5cef8`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:05:06,332::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:05:06,332::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:05:06,332::task::993::Storage.TaskManager.Task::(_decref) Task=`eaa08b33-1a66-4682-9d5f-45ac67a5cef8`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:05:06,361::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:05:06,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:06,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913962 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:08,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:05:08,642::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:05:08,644::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:05:09,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556883 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:11,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:05:11,713::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:05:11,727::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:05:11,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006354 s, 823 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:05:11,746::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000606503 s, 612 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:11,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.590075 s, 1.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:13,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:13,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914896 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:15,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:15,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0969061 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:18,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:18,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914755 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:05:18,654::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:05:18,655::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:05:20,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:20,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918589 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:05:21,374::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:05:21,756::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:05:21,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:05:21,782::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623236 s, 595 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:05:21,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629925 s, 830 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:22,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:22,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913475 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:05:23,024::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:05:24,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:24,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913486 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:26,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:26,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913393 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:05:28,664::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:05:28,665::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:05:28,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:28,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091455 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:30,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:30,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913371 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:05:31,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:05:31,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:05:31,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000607896 s, 610 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:05:31,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000722714 s, 724 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:32,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:33,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915758 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:35,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:35,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543123 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:05:36,389::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:05:37,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:37,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097479 s, 10.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:05:38,674::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:05:38,675::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:05:39,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:39,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922344 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:05:41,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:05:41,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:05:41,849::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641162 s, 579 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:05:41,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626043 s, 835 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:41,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:42,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560314 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:44,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:44,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916291 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:46,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:46,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913633 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:05:48,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:05:48,683::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:05:48,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:48,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913074 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:50,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:50,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914364 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:05:51,401::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:05:51,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:05:51,865::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:05:51,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000739578 s, 502 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:05:51,879::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659111 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:52,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:53,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914069 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:05:53,055::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:05:55,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:55,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091498 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:05:57,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:57,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917267 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:05:58,694::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:05:58,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:05:59,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:05:59,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914216 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:01,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:01,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913929 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:06:01,880::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:06:01,892::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:06:01,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673679 s, 551 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:06:01,906::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00122474 s, 427 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:03,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:03,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914588 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:06,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:06,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912379 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:06:06,404::task::595::Storage.TaskManager.Task::(_updateState) Task=`38c89e0f-8e2f-4b20-a186-f7e0c53337a9`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:06:06,404::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:06:06,407::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:06:06,407::task::1191::Storage.TaskManager.Task::(prepare) Task=`38c89e0f-8e2f-4b20-a186-f7e0c53337a9`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:06:06,407::task::595::Storage.TaskManager.Task::(_updateState) Task=`38c89e0f-8e2f-4b20-a186-f7e0c53337a9`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:06:06,407::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:06:06,407::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:06:06,407::task::993::Storage.TaskManager.Task::(_decref) Task=`38c89e0f-8e2f-4b20-a186-f7e0c53337a9`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:06:06,435::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:06:08,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:08,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914041 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:06:08,697::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:06:08,698::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:06:10,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:10,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55039 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:06:11,908::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:06:11,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:06:11,931::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000823942 s, 450 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:06:11,932::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00108627 s, 481 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:12,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:12,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915987 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:14,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:15,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915806 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:17,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:17,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915762 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:06:18,710::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:06:18,711::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:06:19,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:19,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925842 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:21,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:21,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091445 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:06:21,448::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:06:21,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:06:21,949::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:06:21,950::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:06:21,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000719539 s, 727 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:06:23,094::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:06:23,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:23,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914223 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:25,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:26,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560752 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:28,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:28,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547531 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:06:28,722::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:06:28,723::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:06:30,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:30,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921144 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:06:31,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:06:31,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:06:31,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000631008 s, 588 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:06:32,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00158143 s, 331 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:32,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:32,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915537 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:34,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:35,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914481 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:06:36,461::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:06:37,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:37,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914262 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:06:38,728::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:06:38,730::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:06:39,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:39,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916268 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:41,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:41,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915379 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:06:42,006::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:06:42,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:06:42,018::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683547 s, 543 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:06:42,031::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683511 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:43,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:43,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914099 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:45,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:45,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0984017 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:47,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:47,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912957 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:06:48,741::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:06:48,742::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:06:49,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:50,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913106 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:06:51,474::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:06:52,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:06:52,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:06:52,133::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:06:52,145::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000759328 s, 489 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:06:52,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00118811 s, 440 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:52,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.123827 s, 8.3 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:06:53,145::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:06:54,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:54,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914426 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:56,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:56,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913918 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:06:58,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:06:58,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092451 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:06:58,748::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:06:58,749::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:07:00,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:00,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915815 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:07:02,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:07:02,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639099 s, 581 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:07:02,167::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:07:02,179::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630891 s, 829 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:02,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:02,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091379 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:04,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:04,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912914 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:07:06,477::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7e82d64-4b57-4f06-8aa7-87529d64ef08`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:07:06,477::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:07:06,480::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:07:06,481::task::1191::Storage.TaskManager.Task::(prepare) Task=`c7e82d64-4b57-4f06-8aa7-87529d64ef08`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:07:06,481::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7e82d64-4b57-4f06-8aa7-87529d64ef08`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:07:06,481::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:07:06,481::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:07:06,481::task::993::Storage.TaskManager.Task::(_decref) Task=`c7e82d64-4b57-4f06-8aa7-87529d64ef08`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:07:06,508::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:07:06,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:07,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54966 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:07:08,755::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:07:08,756::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:07:09,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:10,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541675 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:12,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:12,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914298 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:07:12,180::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:07:12,191::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:07:12,202::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659006 s, 563 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:07:12,204::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000918413 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:14,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:14,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091582 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:16,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:16,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0974011 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:18,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:18,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914379 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:07:18,763::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:07:18,764::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:07:20,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:20,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913978 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:07:21,521::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:07:22,208::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:07:22,219::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:07:22,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00543745 s, 68.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:07:22,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672425 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:22,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:22,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913284 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:07:23,187::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:07:24,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:24,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913787 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:26,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:27,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915461 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:07:28,775::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:07:28,776::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:07:29,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:29,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913714 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:31,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:31,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916085 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:07:32,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:07:32,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:07:32,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693565 s, 535 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:07:32,264::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658296 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:33,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:33,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:35,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:35,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914296 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:07:36,533::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:07:37,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:37,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913097 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:07:38,788::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:07:38,789::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:07:39,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:40,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515467 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:07:42,268::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:07:42,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:07:42,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000939905 s, 395 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:07:42,293::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651517 s, 803 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:42,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:42,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914492 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:44,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:44,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913818 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:46,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:46,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912922 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:48,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:07:48,800::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:07:48,801::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:07:48,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914715 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:50,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:51,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543278 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:07:51,549::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 16:07:52,296::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 16:07:52,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:07:52,312::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000760016 s, 488 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:07:52,329::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:07:52,330::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:07:52,333::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:07:52,334::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:07:52,334::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:07:52,335::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:07:52,336::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:07:52,347::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641267 s, 816 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:07:53,221::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:07:53,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:53,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914912 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:55,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:55,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913933 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:07:57,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:57,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914349 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:07:58,811::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:07:58,812::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:07:59,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:07:59,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915182 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:01,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:01,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913571 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:08:02,324::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:08:02,336::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629815 s, 589 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:08:02,351::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:08:02,362::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656963 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:04,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:04,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916396 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:06,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:08:06,552::task::595::Storage.TaskManager.Task::(_updateState) Task=`53f58b4f-665c-4bd5-ba78-8a8714829880`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:08:06,552::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:08:06,644::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:08:06,645::task::1191::Storage.TaskManager.Task::(prepare) Task=`53f58b4f-665c-4bd5-ba78-8a8714829880`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:08:06,645::task::595::Storage.TaskManager.Task::(_updateState) Task=`53f58b4f-665c-4bd5-ba78-8a8714829880`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:08:06,646::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:08:06,646::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:08:06,647::task::993::Storage.TaskManager.Task::(_decref) Task=`53f58b4f-665c-4bd5-ba78-8a8714829880`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 16:08:06,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.51092 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:08:06,677::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:08:08,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:08,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915141 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:08:08,821::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:08:08,822::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:08:10,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:10,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915555 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:08:12,339::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:08:12,351::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622512 s, 596 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:08:12,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:08:12,377::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683331 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:12,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:13,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917732 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:15,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:15,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917758 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:17,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:17,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091555 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:08:18,833::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:08:18,834::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:08:19,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:19,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914724 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:21,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:21,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914654 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:08:21,689::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:08:22,353::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:08:22,368::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:08:22,369::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:08:22,372::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:08:22,373::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:08:22,374::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:08:22,374::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:08:22,375::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:08:22,387::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622876 s, 596 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:08:22,390::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:08:22,406::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00118807 s, 440 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:08:23,255::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:08:23,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:23,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914723 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:25,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:26,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558333 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:28,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:28,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918902 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:08:28,845::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:08:28,846::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:08:30,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:30,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916302 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:08:32,404::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:08:32,416::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:08:32,432::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000770113 s, 482 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:08:32,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687737 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:32,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:32,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915276 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:34,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:34,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914582 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:36,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:08:36,703::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:08:36,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914395 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:38,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:08:38,858::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:08:38,859::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:08:38,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915767 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:40,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:41,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915062 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:08:42,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:08:42,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690811 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:08:42,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:08:42,466::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000622588 s, 840 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:43,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:43,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913911 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:45,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:45,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.56153 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:47,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:47,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919879 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:08:48,870::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:08:48,871::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:08:50,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:50,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544153 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:08:51,714::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:08:52,467::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:08:52,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000753337 s, 492 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:08:52,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:08:52,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655571 s, 798 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:52,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:08:53,288::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:08:53,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37045 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:55,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:56,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0942316 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:08:58,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:08:58,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915263 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:08:58,880::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:08:58,881::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:09:00,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:00,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911658 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:02,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:02,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916471 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:09:02,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:09:02,506::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:09:02,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000603275 s, 615 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:09:02,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000620924 s, 842 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:04,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:04,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911914 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:06,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:06,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912865 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:09:06,717::task::595::Storage.TaskManager.Task::(_updateState) Task=`faf5fc83-4ed3-4265-accc-512a27c7d341`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:09:06,718::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:09:06,720::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:09:06,720::task::1191::Storage.TaskManager.Task::(prepare) Task=`faf5fc83-4ed3-4265-accc-512a27c7d341`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:09:06,720::task::595::Storage.TaskManager.Task::(_updateState) Task=`faf5fc83-4ed3-4265-accc-512a27c7d341`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:09:06,721::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:09:06,721::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:09:06,721::task::993::Storage.TaskManager.Task::(_decref) Task=`faf5fc83-4ed3-4265-accc-512a27c7d341`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:09:06,746::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:09:08,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:08,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913936 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:09:08,890::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:09:08,892::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:09:10,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:10,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915053 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:09:12,522::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:09:12,533::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:09:12,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00061733 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:09:12,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000748958 s, 698 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:12,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:13,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917114 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:15,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:15,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091568 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:17,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:17,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918507 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:09:18,900::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:09:18,901::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:09:19,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:19,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914428 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:21,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:21,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915456 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:09:21,758::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:09:22,551::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:09:22,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:09:22,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000711304 s, 522 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:09:22,575::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000929493 s, 563 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:09:23,324::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:09:23,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:23,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915812 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:25,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:26,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964547 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:28,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:28,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912446 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:09:28,911::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:09:28,912::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:09:30,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:30,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915074 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:32,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:32,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913588 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:09:32,579::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:09:32,590::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:09:32,601::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000793991 s, 467 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:09:32,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000612684 s, 854 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:34,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:34,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550007 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:09:36,769::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:09:37,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:37,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912247 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:09:38,919::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:09:38,920::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:09:39,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:39,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504725 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:41,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:41,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916561 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:09:42,607::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:09:42,619::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:09:42,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647487 s, 573 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:09:42,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671664 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:43,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:43,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914162 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:45,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:46,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0997007 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:48,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:48,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914074 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:09:48,931::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:09:48,932::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:09:50,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:50,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913601 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:09:51,781::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:09:52,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:09:52,927::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:09:52,939::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:09:52,955::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674645 s, 550 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:09:52,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684888 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:52,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.596731 s, 1.7 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:09:53,358::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:09:54,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:55,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913106 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:09:57,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:57,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913838 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:09:58,939::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:09:58,940::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:09:59,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:09:59,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913481 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:01,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:01,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091364 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:10:02,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:10:02,976::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:10:02,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00415708 s, 89.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:10:02,989::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657225 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:03,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:03,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915173 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:05,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:05,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914755 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:10:06,784::task::595::Storage.TaskManager.Task::(_updateState) Task=`8fdcde15-75de-4e83-98bc-144f8c5f3507`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:10:06,784::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:10:06,787::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:10:06,787::task::1191::Storage.TaskManager.Task::(prepare) Task=`8fdcde15-75de-4e83-98bc-144f8c5f3507`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:10:06,787::task::595::Storage.TaskManager.Task::(_updateState) Task=`8fdcde15-75de-4e83-98bc-144f8c5f3507`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:10:06,787::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:10:06,787::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:10:06,788::task::993::Storage.TaskManager.Task::(_decref) Task=`8fdcde15-75de-4e83-98bc-144f8c5f3507`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:10:06,814::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:10:07,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:10:08,951::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:10:08,953::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:10:09,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.35393 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:11,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:11,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.094801 s, 10.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:10:12,993::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:10:13,004::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:10:13,020::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641691 s, 578 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:10:13,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.001415 s, 370 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:13,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:13,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913445 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:15,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:15,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916753 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:17,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:17,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917418 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:10:18,964::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:10:18,965::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:10:19,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:19,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915264 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:21,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:21,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914138 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:10:21,825::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:10:23,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:10:23,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648302 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:10:23,040::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:10:23,053::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648272 s, 807 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:10:23,391::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:10:23,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:23,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913998 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:25,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:26,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091388 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:28,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:28,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912311 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:10:28,976::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:10:28,977::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:10:30,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:30,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:32,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:32,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916614 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:10:33,054::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:10:33,066::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:10:33,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000698455 s, 531 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:10:33,078::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000856743 s, 610 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:34,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:34,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915823 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:36,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:36,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920736 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:10:36,837::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:10:38,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:38,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916689 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:10:38,990::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:10:38,991::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:10:40,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:40,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927318 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:42,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:10:43,096::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:10:43,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:10:43,125::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658401 s, 563 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:10:43,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691103 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:43,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.131707 s, 7.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:45,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:45,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916516 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:47,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:47,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915557 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:10:49,001::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:10:49,002::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:10:49,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:49,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914166 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:51,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:51,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913653 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:10:51,849::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:10:53,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:10:53,147::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651607 s, 569 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:10:53,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:10:53,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669478 s, 781 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:10:53,423::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:10:53,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:53,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914367 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:55,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:55,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914125 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:10:57,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:10:57,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915097 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:10:59,013::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:10:59,014::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:10:59,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:00,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913328 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:02,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:02,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914757 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:11:03,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:11:03,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:11:03,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00470364 s, 78.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:11:03,190::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674329 s, 776 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:04,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:04,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913792 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:06,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:06,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915299 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:11:06,852::task::595::Storage.TaskManager.Task::(_updateState) Task=`86f2f2f8-918b-4c94-9259-058087bd079a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:11:06,853::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:11:06,855::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:11:06,856::task::1191::Storage.TaskManager.Task::(prepare) Task=`86f2f2f8-918b-4c94-9259-058087bd079a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:11:06,856::task::595::Storage.TaskManager.Task::(_updateState) Task=`86f2f2f8-918b-4c94-9259-058087bd079a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:11:06,856::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:11:06,856::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:11:06,856::task::993::Storage.TaskManager.Task::(_decref) Task=`86f2f2f8-918b-4c94-9259-058087bd079a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:11:06,882::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:11:08,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:08,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912485 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:11:09,020::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:11:09,021::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:11:10,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:10,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911986 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:12,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:12,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915535 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:11:13,193::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:11:13,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:11:13,219::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637481 s, 582 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:11:13,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000978934 s, 534 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:14,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:14,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091412 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:16,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:17,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913424 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:19,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:11:19,041::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:11:19,042::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:11:19,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915908 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:21,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:21,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558499 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:11:21,894::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:11:23,223::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:11:23,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:11:23,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000796498 s, 466 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:11:23,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000786108 s, 665 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:11:23,457::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:11:23,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:24,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546233 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:26,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:26,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504755 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:28,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:11:29,038::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:11:29,039::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:11:29,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541236 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:31,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:31,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913153 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:11:33,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:11:33,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:11:33,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000700486 s, 530 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:11:33,276::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00109908 s, 476 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:33,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:33,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912532 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:35,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:35,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915408 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:11:36,905::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:11:37,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:37,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913306 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:11:39,047::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:11:39,048::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:11:39,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:39,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913734 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:41,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:42,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919299 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:11:43,277::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:11:43,289::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:11:43,304::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000732664 s, 506 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:11:43,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000837164 s, 625 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:44,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:44,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914577 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:46,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:46,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970726 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:48,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:48,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091327 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:11:49,056::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:11:49,057::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:11:50,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:50,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913496 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:11:51,922::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:11:52,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:52,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915016 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:11:53,308::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:11:53,321::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:11:53,322::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000722086 s, 514 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:11:53,334::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000770456 s, 679 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:11:53,491::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:11:54,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:54,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914631 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:56,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:11:57,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913368 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:11:59,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:11:59,069::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:11:59,070::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:11:59,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913862 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:01,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:02,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39866 s, 732 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:12:03,338::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:12:03,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:12:03,360::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000733599 s, 506 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:12:03,362::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069045 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:04,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:05,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547624 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:12:06,925::task::595::Storage.TaskManager.Task::(_updateState) Task=`e7a22885-4925-4c26-8655-6eec03f7934a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:12:06,925::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:12:06,928::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:12:06,928::task::1191::Storage.TaskManager.Task::(prepare) Task=`e7a22885-4925-4c26-8655-6eec03f7934a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:12:06,929::task::595::Storage.TaskManager.Task::(_updateState) Task=`e7a22885-4925-4c26-8655-6eec03f7934a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:12:06,929::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:12:06,929::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:12:06,929::task::993::Storage.TaskManager.Task::(_decref) Task=`e7a22885-4925-4c26-8655-6eec03f7934a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:12:06,956::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:12:07,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:07,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914708 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:12:09,081::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:12:09,082::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:12:09,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:09,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915583 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:11,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:11,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915284 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:12:13,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:12:13,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:12:13,387::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00543851 s, 68.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:12:13,388::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000701264 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:13,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:13,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091484 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:15,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:15,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926725 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:17,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:17,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910883 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:12:19,089::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:12:19,090::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:12:19,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:20,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914383 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:12:21,968::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:12:22,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:22,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915153 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:12:23,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:12:23,404::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:12:23,414::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0060684 s, 61.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:12:23,417::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000701575 s, 745 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:12:23,522::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:12:24,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:24,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913727 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:26,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:26,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914813 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:28,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:28,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092005 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:12:29,101::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:12:29,102::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:12:30,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:30,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914373 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:32,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:32,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091415 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:12:33,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:12:33,431::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000710363 s, 522 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:12:33,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:12:33,445::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703097 s, 744 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:34,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:34,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915562 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:36,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:12:36,982::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:12:37,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914865 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:39,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:12:39,114::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:12:39,115::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:12:39,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541463 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:41,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:41,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913965 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:12:43,447::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:12:43,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:12:43,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684317 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:12:43,475::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635336 s, 823 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:43,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:43,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913829 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:45,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:45,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0972836 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:48,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:48,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916132 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:12:49,126::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:12:49,127::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:12:50,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:50,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091499 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:12:51,993::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:12:52,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:52,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913549 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:12:53,478::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 16:12:53,478::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:12:53,496::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683523 s, 543 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:12:53,511::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:12:53,512::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:12:53,515::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:12:53,516::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:12:53,517::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:12:53,517::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:12:53,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:12:53,529::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688371 s, 760 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:12:53,553::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:12:54,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:54,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914707 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:56,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:56,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914862 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:12:58,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:12:58,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914952 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:12:59,135::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:12:59,136::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:13:00,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:00,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914965 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:02,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:03,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915656 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:13:03,507::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:13:03,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00145339 s, 255 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:13:03,533::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:13:03,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660952 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:05,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:05,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913971 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:13:06,996::task::595::Storage.TaskManager.Task::(_updateState) Task=`f3af3f40-b33f-474a-b972-f45b580d7ad5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:13:06,996::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:13:06,999::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:13:06,999::task::1191::Storage.TaskManager.Task::(prepare) Task=`f3af3f40-b33f-474a-b972-f45b580d7ad5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:13:06,999::task::595::Storage.TaskManager.Task::(_updateState) Task=`f3af3f40-b33f-474a-b972-f45b580d7ad5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:13:07,000::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:13:07,000::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:13:07,000::task::993::Storage.TaskManager.Task::(_decref) Task=`f3af3f40-b33f-474a-b972-f45b580d7ad5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:13:07,026::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:13:07,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:07,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915696 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:13:09,143::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:13:09,145::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:13:09,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:09,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914439 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:11,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:11,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502954 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:13:13,523::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:13:13,535::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649062 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:13:13,548::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:13:13,559::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650226 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:13,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:14,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541473 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:16,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:17,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510744 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:19,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:19,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915472 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:13:19,156::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:13:19,157::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:13:21,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:21,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913465 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:13:22,045::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:13:23,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:23,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913779 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:13:23,537::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:13:23,551::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:13:23,552::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:13:23,555::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:13:23,557::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:13:23,557::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:13:23,557::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:13:23,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:13:23,571::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639929 s, 580 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:13:23,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:13:23,586::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631125 s, 829 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:13:23,587::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:13:25,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:25,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913581 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:27,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:27,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917269 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:13:29,164::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:13:29,165::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:13:29,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:29,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910316 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:31,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:31,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914365 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:13:33,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:13:33,599::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:13:33,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643676 s, 576 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:13:33,616::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684666 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:33,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:33,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913559 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:35,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:36,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913539 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:13:37,057::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:13:38,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:38,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914691 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:13:39,176::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:13:39,177::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:13:40,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:40,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915568 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:42,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:42,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913922 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:13:43,621::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:13:43,632::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:13:43,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000728649 s, 509 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:13:43,645::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675776 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:44,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:44,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914284 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:46,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:46,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919552 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:48,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:48,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914597 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:13:49,188::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:13:49,189::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:13:50,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:50,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913945 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:13:52,071::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:13:52,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:53,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.534968 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:13:53,642::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 16:13:53,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:13:53,659::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:13:53,659::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000738754 s, 502 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:13:53,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641595 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:55,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:55,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915201 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:13:57,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:57,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913096 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:13:59,195::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:13:59,196::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:13:59,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:13:59,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913297 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:01,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:01,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914989 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:14:03,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:14:03,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:14:03,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0056336 s, 65.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:14:03,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703397 s, 744 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:03,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:04,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915273 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:06,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:06,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.492858 s, 2.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:14:07,075::task::595::Storage.TaskManager.Task::(_updateState) Task=`f006080e-f967-41c9-8c7c-a35fcd681714`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:14:07,075::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:14:07,078::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:14:07,078::task::1191::Storage.TaskManager.Task::(prepare) Task=`f006080e-f967-41c9-8c7c-a35fcd681714`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:14:07,079::task::595::Storage.TaskManager.Task::(_updateState) Task=`f006080e-f967-41c9-8c7c-a35fcd681714`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:14:07,079::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:14:07,079::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:14:07,079::task::993::Storage.TaskManager.Task::(_decref) Task=`f006080e-f967-41c9-8c7c-a35fcd681714`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:14:07,108::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:14:08,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:08,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915905 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:14:09,207::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:14:09,208::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:14:10,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:10,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914399 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:12,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:13,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539837 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:14:13,706::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:14:13,717::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:14:13,729::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00526643 s, 70.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:14:13,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700672 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:15,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:15,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915436 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:17,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:18,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548319 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:14:19,219::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:14:19,220::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:14:20,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:20,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913674 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:14:22,120::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:14:22,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:22,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913816 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:14:23,676::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 16:14:23,735::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:14:23,749::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662948 s, 560 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:14:23,748::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:14:23,763::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632705 s, 827 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:24,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:24,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913456 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:26,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:26,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914925 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:28,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:28,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914598 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:14:29,226::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:14:29,227::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:14:30,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:30,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132806 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:32,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:32,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914755 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:14:33,763::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:14:33,775::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:14:33,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000842024 s, 441 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:14:33,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000623057 s, 839 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:34,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:35,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914112 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:37,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:14:37,134::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:14:37,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916213 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:39,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:14:39,236::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:14:39,238::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:14:39,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913189 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:41,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:41,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915895 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:43,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:43,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091779 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:14:43,791::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:14:43,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:14:43,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657532 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:14:43,819::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651559 s, 803 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:45,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:45,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999545 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:47,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:48,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542758 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:14:49,246::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:14:49,247::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:14:50,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:50,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915154 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:14:52,147::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:14:52,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:52,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915881 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:14:53,712::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 16:14:53,823::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:14:53,834::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:14:53,845::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000822914 s, 451 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:14:53,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686422 s, 762 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:54,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:54,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912685 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:56,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:56,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912748 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:14:58,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:14:58,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926589 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:14:59,258::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:14:59,259::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:15:00,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:02,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.35774 s, 754 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:15:03,850::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:15:03,862::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:15:03,874::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00523032 s, 70.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:15:03,875::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000740871 s, 706 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:04,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:04,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5411 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:06,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:07,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918338 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:15:07,150::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5d7227d-0ca9-49b2-b6ed-d7455d526926`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:15:07,150::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:15:07,154::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:15:07,154::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5d7227d-0ca9-49b2-b6ed-d7455d526926`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:15:07,154::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5d7227d-0ca9-49b2-b6ed-d7455d526926`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:15:07,154::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:15:07,154::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:15:07,154::task::993::Storage.TaskManager.Task::(_decref) Task=`a5d7227d-0ca9-49b2-b6ed-d7455d526926`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:15:07,180::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:15:09,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:09,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915459 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:15:09,270::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:15:09,271::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:15:11,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:12,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.33146 s, 769 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:15:13,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:15:13,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:15:13,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000607841 s, 610 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:15:13,902::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651047 s, 803 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:14,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:14,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916018 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:16,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:17,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548712 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:19,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:15:19,280::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:15:19,281::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:15:19,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54107 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:21,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:15:22,191::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:15:22,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551507 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:15:23,743::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 16:15:23,907::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:15:23,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:15:23,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685935 s, 541 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:15:23,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637814 s, 820 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:24,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:24,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544082 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:26,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:27,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915015 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:29,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:15:29,287::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:15:29,288::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:15:29,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541478 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:31,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:32,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54788 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:15:33,935::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:15:33,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:15:33,949::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000726435 s, 511 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:15:33,963::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0011873 s, 440 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:34,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:34,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914326 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:36,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:36,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545615 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:15:37,203::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:15:38,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:38,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918994 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:15:39,298::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:15:39,299::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:15:40,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:41,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915416 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:43,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:43,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917585 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:15:44,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:15:44,026::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:15:44,052::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000619682 s, 599 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:15:44,054::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658642 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:45,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:45,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912361 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:47,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:47,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912988 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:15:49,310::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:15:49,311::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:15:49,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:49,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914238 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:51,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:51,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918206 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:15:52,217::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:15:53,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:15:53,776::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:15:53,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919496 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:15:54,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:15:54,060::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:15:54,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006441 s, 576 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:15:54,081::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070439 s, 742 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:55,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:56,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559814 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:15:58,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:15:58,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559723 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:15:59,320::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:15:59,321::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:16:00,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:01,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559795 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:03,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:03,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914634 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:16:04,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:16:04,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:16:04,111::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688197 s, 760 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:16:04,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000770667 s, 481 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:05,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:05,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914799 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:16:07,220::task::595::Storage.TaskManager.Task::(_updateState) Task=`e859d106-65a4-4e21-aa82-2f00f061bc33`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:16:07,220::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:16:07,223::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:16:07,223::task::1191::Storage.TaskManager.Task::(prepare) Task=`e859d106-65a4-4e21-aa82-2f00f061bc33`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:16:07,223::task::595::Storage.TaskManager.Task::(_updateState) Task=`e859d106-65a4-4e21-aa82-2f00f061bc33`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:16:07,223::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:16:07,224::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:16:07,224::task::993::Storage.TaskManager.Task::(_decref) Task=`e859d106-65a4-4e21-aa82-2f00f061bc33`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:16:07,251::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:16:07,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:07,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091233 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:16:09,332::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:16:09,334::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:16:09,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:10,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913669 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:12,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:12,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507411 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:16:14,121::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:16:14,121::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:16:14,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716973 s, 729 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:16:14,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618472 s, 600 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:14,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:14,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914872 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:16,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:16,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914239 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:18,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:18,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913511 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:16:19,345::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:16:19,346::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:16:20,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:21,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914327 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:16:22,264::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:16:23,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:23,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091444 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:16:23,810::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 16:16:24,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:16:24,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:16:24,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000718723 s, 516 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:16:24,178::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000589805 s, 887 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:25,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:25,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910613 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:27,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:27,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913808 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:16:29,354::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:16:29,355::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:16:29,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:29,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920946 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:31,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:32,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913367 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:34,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:16:34,188::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:16:34,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678597 s, 547 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:16:34,204::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:16:34,220::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664633 s, 787 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:34,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129344 s, 7.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:36,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:36,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914978 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:16:37,276::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:16:38,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:38,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913942 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:16:39,366::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:16:39,367::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:16:40,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:40,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922248 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:42,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:42,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914231 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:16:44,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:16:44,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:16:44,234::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000745223 s, 498 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:16:44,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000871739 s, 600 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:44,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:44,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919842 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:46,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:46,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913975 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:48,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:16:49,376::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:16:49,377::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:16:49,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546079 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:51,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:51,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916684 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:16:52,289::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:16:53,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:53,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0978889 s, 10.5 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:16:53,850::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 16:16:54,249::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:16:54,261::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:16:54,272::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000615559 s, 603 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:16:54,274::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648283 s, 807 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:55,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:55,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913853 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:16:57,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:16:57,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091281 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:16:59,388::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:16:59,390::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:16:59,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:00,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916445 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:02,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:02,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912522 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:04,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:17:04,335::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:17:04,346::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:17:04,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000731454 s, 715 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:17:04,362::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000805208 s, 461 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:04,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.127325 s, 8.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:06,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:06,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554175 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:17:07,292::task::595::Storage.TaskManager.Task::(_updateState) Task=`9fe39c02-7b16-4616-98f3-90c455788ca1`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:17:07,292::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:17:07,295::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:17:07,295::task::1191::Storage.TaskManager.Task::(prepare) Task=`9fe39c02-7b16-4616-98f3-90c455788ca1`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:17:07,295::task::595::Storage.TaskManager.Task::(_updateState) Task=`9fe39c02-7b16-4616-98f3-90c455788ca1`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:17:07,295::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:17:07,296::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:17:07,296::task::993::Storage.TaskManager.Task::(_decref) Task=`9fe39c02-7b16-4616-98f3-90c455788ca1`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:17:07,323::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:17:08,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:09,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914401 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:17:09,398::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:17:09,399::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:17:11,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:11,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915181 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:13,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:13,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914192 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:17:14,370::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:17:14,381::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:17:14,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617174 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:17:14,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664119 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:15,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:15,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914191 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:17,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:17,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916572 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:17:19,410::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:17:19,411::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:17:19,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:19,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091587 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:21,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:21,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914685 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:17:22,340::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:17:23,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:17:23,884::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:17:23,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924943 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:17:24,399::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:17:24,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:17:24,413::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000615008 s, 603 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:17:24,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000740261 s, 707 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:25,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:26,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914403 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:28,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:28,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914222 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:17:29,434::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:17:29,435::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:17:30,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:30,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545241 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:32,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:32,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913497 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:17:34,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:17:34,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:17:34,453::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00389875 s, 95.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:17:34,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0014839 s, 352 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:34,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:34,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912249 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:36,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:37,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913686 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:17:37,353::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:17:39,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:39,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0971231 s, 10.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:17:39,448::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:17:39,450::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:17:41,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:41,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913188 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:43,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:43,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.533034 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:17:44,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:17:44,469::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:17:44,486::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620653 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:17:44,488::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665078 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:45,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:45,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0969884 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:48,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:48,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912236 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:17:49,461::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:17:49,462::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:17:50,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:50,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916157 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:52,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:17:52,376::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:17:52,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913789 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:17:53,916::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 16:17:54,493::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 16:17:54,495::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:17:54,515::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000700449 s, 530 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:17:54,528::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:17:54,529::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:17:54,532::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:17:54,533::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:17:54,534::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:17:54,534::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:17:54,535::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:17:54,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663845 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:54,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:54,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920862 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:56,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:56,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912981 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:17:58,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:17:58,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913485 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:17:59,468::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:17:59,469::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:18:00,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:01,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913506 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:03,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:03,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.176483 s, 5.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:18:04,524::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:18:04,536::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664777 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:18:04,550::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:18:04,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687758 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:05,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:05,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913166 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:07,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:18:07,383::task::595::Storage.TaskManager.Task::(_updateState) Task=`b5ed1b68-ffe0-49f5-a738-4a13e04ce469`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:18:07,384::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:18:07,910::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:18:07,911::task::1191::Storage.TaskManager.Task::(prepare) Task=`b5ed1b68-ffe0-49f5-a738-4a13e04ce469`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:18:07,912::task::595::Storage.TaskManager.Task::(_updateState) Task=`b5ed1b68-ffe0-49f5-a738-4a13e04ce469`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:18:07,912::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:18:07,913::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:18:07,914::task::993::Storage.TaskManager.Task::(_decref) Task=`b5ed1b68-ffe0-49f5-a738-4a13e04ce469`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 16:18:07,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5496 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:18:07,947::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:18:09,484::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:18:09,485::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:18:09,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:10,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.501068 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:12,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:13,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547182 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:18:14,540::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:18:14,552::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065564 s, 566 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:18:14,565::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:18:14,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000715668 s, 731 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:15,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:15,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913002 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:17,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:17,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913362 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:19,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:19,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912726 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:18:19,496::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:18:19,497::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:18:21,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:21,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091223 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:18:22,959::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:18:23,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:23,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912683 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:18:23,957::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 16:18:24,554::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:18:24,568::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:18:24,570::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:18:24,573::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:18:24,574::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:18:24,575::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:18:24,575::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:18:24,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:18:24,588::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000640855 s, 579 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:18:24,591::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:18:24,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621222 s, 842 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:25,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:25,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912782 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:27,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:27,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913159 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:18:29,505::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:18:29,507::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:18:29,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:29,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914963 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:31,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:32,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913087 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:34,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:18:34,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:18:34,639::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:18:34,640::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697154 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:18:34,658::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692143 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:34,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.565608 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:36,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:36,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914087 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:18:37,973::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:18:38,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:38,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913296 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:18:39,515::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:18:39,516::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:18:40,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:41,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091431 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:43,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:43,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913843 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:18:44,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:18:44,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:18:44,673::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00075689 s, 490 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:18:44,686::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671453 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:45,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:45,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914354 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:47,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:47,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555892 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:18:49,527::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:18:49,528::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:18:49,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:49,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913119 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:51,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:52,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913548 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:18:52,989::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:18:53,988::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:18:54,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:54,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09142 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:18:54,689::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:18:54,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:18:54,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00057499 s, 645 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:18:54,714::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00063997 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:56,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:56,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912487 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:18:58,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:18:58,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914505 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:18:59,535::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:18:59,536::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:19:00,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:00,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510771 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:02,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:03,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915185 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:19:04,720::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:19:04,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:19:04,741::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000706891 s, 525 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:19:04,742::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664973 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:05,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:05,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911204 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:07,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:07,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914059 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:19:07,992::task::595::Storage.TaskManager.Task::(_updateState) Task=`821eb5e8-2733-4d65-9a9a-0c402f5f22db`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:19:07,992::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:19:07,995::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:19:07,995::task::1191::Storage.TaskManager.Task::(prepare) Task=`821eb5e8-2733-4d65-9a9a-0c402f5f22db`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:19:07,995::task::595::Storage.TaskManager.Task::(_updateState) Task=`821eb5e8-2733-4d65-9a9a-0c402f5f22db`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:19:07,996::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:19:07,996::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:19:07,996::task::993::Storage.TaskManager.Task::(_decref) Task=`821eb5e8-2733-4d65-9a9a-0c402f5f22db`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:19:08,024::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:19:09,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:09,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911825 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:19:09,548::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:19:09,549::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:19:11,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:11,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913978 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:13,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:14,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549164 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:19:14,746::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:19:14,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:19:14,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000725109 s, 512 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:19:14,771::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626202 s, 835 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:16,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:16,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100044 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:18,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:18,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909477 s, 11.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:19:19,560::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:19:19,561::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:19:20,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:20,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912767 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:22,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:22,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913968 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:19:23,036::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:19:24,019::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:19:24,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:19:24,772::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:19:24,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000717048 s, 517 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:19:24,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:19:24,881::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000617101 s, 848 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:24,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.118381 s, 8.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:26,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:27,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.532717 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:29,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:19:29,572::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:19:29,574::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:19:29,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50445 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:31,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:32,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546302 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:34,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:34,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912474 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:19:34,881::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:19:34,893::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:19:34,908::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000753088 s, 493 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:19:34,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643119 s, 813 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:36,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:36,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915518 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:19:38,048::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:19:38,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:38,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091444 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:19:39,584::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:19:39,586::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:19:40,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:41,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913548 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:43,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:43,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912847 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:19:44,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:19:44,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:19:44,937::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675471 s, 549 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:19:44,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641838 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:45,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:45,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917041 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:47,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:47,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913864 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:49,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:49,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912783 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:19:49,594::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:19:49,595::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:19:51,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:52,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50969 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:19:53,063::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:19:54,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:19:54,056::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:19:54,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091311 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:19:54,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:19:54,954::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:19:54,955::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688182 s, 539 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:19:54,968::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000607081 s, 861 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:56,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:56,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913099 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:19:58,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:19:58,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914658 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:19:59,606::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:19:59,607::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:20:00,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:00,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914449 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:02,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:02,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914394 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:04,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:04,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091386 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:20:04,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:20:04,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:20:04,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623845 s, 595 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:20:04,995::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692982 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:06,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:06,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913173 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:20:08,066::task::595::Storage.TaskManager.Task::(_updateState) Task=`754f3437-cfd6-432e-9892-ea3b70172deb`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:20:08,066::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:20:08,069::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:20:08,069::task::1191::Storage.TaskManager.Task::(prepare) Task=`754f3437-cfd6-432e-9892-ea3b70172deb`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:20:08,070::task::595::Storage.TaskManager.Task::(_updateState) Task=`754f3437-cfd6-432e-9892-ea3b70172deb`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:20:08,070::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:20:08,070::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:20:08,070::task::993::Storage.TaskManager.Task::(_decref) Task=`754f3437-cfd6-432e-9892-ea3b70172deb`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:20:08,095::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:20:08,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:08,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911759 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:20:09,610::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:20:09,611::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:20:10,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:11,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914143 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:13,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:13,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545599 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:20:15,000::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:20:15,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:20:15,022::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691162 s, 537 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:20:15,023::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000768628 s, 680 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:15,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:17,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39156 s, 736 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:19,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:20:19,622::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:20:19,623::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:20:19,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54762 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:21,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:22,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913003 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:20:23,109::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:20:24,089::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:20:24,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:24,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913828 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:20:25,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:20:25,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:20:25,051::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636891 s, 583 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:20:25,053::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651886 s, 802 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:26,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:26,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914159 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:28,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:28,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926161 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:20:29,635::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:20:29,636::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:20:30,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:30,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914784 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:32,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:33,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541744 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:20:35,056::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:20:35,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:20:35,079::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00452057 s, 82.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:20:35,081::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646103 s, 809 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:35,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:35,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911343 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:37,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:37,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913156 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:20:38,121::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:20:39,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:39,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913623 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:20:39,642::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:20:39,643::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:20:41,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:41,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913959 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:43,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:43,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914612 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:20:45,082::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:20:45,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:20:45,106::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0034696 s, 151 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:20:45,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690438 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:45,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:46,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549794 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:48,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:48,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910911 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:20:49,654::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:20:49,655::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:20:50,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:50,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917755 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:52,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:52,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913359 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:20:53,135::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:20:54,122::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:20:54,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:54,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911873 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:20:55,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:20:55,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:20:55,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00381694 s, 97.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:20:55,136::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680948 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:56,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:56,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091498 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:20:58,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:20:59,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913601 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:20:59,666::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:20:59,667::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:21:01,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:01,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913705 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:03,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:03,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927549 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:21:05,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:21:05,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:21:05,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000604488 s, 614 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:21:05,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000969781 s, 539 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:05,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:06,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548702 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:08,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:21:08,140::task::595::Storage.TaskManager.Task::(_updateState) Task=`d0cea215-0f57-4300-bbb2-3ce8e22f1963`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:21:08,141::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:21:08,154::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:21:08,155::task::1191::Storage.TaskManager.Task::(prepare) Task=`d0cea215-0f57-4300-bbb2-3ce8e22f1963`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:21:08,156::task::595::Storage.TaskManager.Task::(_updateState) Task=`d0cea215-0f57-4300-bbb2-3ce8e22f1963`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:21:08,156::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:21:08,157::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:21:08,159::task::993::Storage.TaskManager.Task::(_decref) Task=`d0cea215-0f57-4300-bbb2-3ce8e22f1963`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 16:21:08,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0958576 s, 10.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:21:08,193::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:21:09,675::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:21:09,676::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:21:10,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:10,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925373 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:12,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:12,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091166 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:14,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:14,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091521 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:21:15,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:21:15,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:21:15,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657144 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:21:15,227::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062603 s, 835 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:16,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:17,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36214 s, 752 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:21:19,688::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:21:19,689::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:21:19,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:20,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913783 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:22,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:22,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912572 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:21:23,206::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:21:24,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:21:24,176::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:21:24,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912769 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:21:25,231::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:21:25,242::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:21:25,253::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689965 s, 538 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:21:25,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000765209 s, 683 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:26,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:27,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.3848 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:29,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:21:29,697::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:21:29,700::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:21:30,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540511 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:32,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:32,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913538 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:34,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:34,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546526 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:21:35,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:21:35,271::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:21:35,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00552754 s, 67.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:21:35,283::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641008 s, 816 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:36,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:37,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913494 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:21:38,219::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:21:39,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:39,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912377 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:21:39,706::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:21:39,707::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:21:41,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:41,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542851 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:43,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:43,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132845 s, 7.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:21:45,287::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:21:45,299::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:21:45,314::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687352 s, 540 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:21:45,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00104023 s, 503 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:45,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:46,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916078 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:48,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:48,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554631 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:21:49,709::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:21:49,710::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:21:50,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:51,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551776 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:21:53,231::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:21:53,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:53,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914206 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:21:54,209::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 16:21:55,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:21:55,330::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:21:55,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00075021 s, 495 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:21:55,343::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00121688 s, 430 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:55,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:55,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912626 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:21:57,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:21:58,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547135 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:21:59,721::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:21:59,722::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:22:00,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:00,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912055 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:02,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:02,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912512 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:04,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:04,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911209 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:22:05,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:22:05,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:22:05,374::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000615255 s, 850 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:22:05,377::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673457 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:06,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:06,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916103 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:22:08,234::task::595::Storage.TaskManager.Task::(_updateState) Task=`66c58af6-5afa-4226-a2b3-b06391f2648b`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:22:08,234::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:22:08,238::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:22:08,238::task::1191::Storage.TaskManager.Task::(prepare) Task=`66c58af6-5afa-4226-a2b3-b06391f2648b`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:22:08,238::task::595::Storage.TaskManager.Task::(_updateState) Task=`66c58af6-5afa-4226-a2b3-b06391f2648b`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:22:08,238::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:22:08,238::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:22:08,239::task::993::Storage.TaskManager.Task::(_decref) Task=`66c58af6-5afa-4226-a2b3-b06391f2648b`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:22:08,263::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:22:08,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:08,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912908 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:22:09,729::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:22:09,730::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:22:10,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:11,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50387 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:13,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:13,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.517941 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:22:15,383::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:22:15,394::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:22:15,406::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069272 s, 755 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:22:15,408::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0010579 s, 351 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:15,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:15,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999218 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:18,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:19,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915355 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:22:19,736::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:22:19,737::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:22:21,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:21,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915728 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:22:23,275::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:22:23,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:23,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912998 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:22:24,242::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 16:22:25,412::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:22:25,424::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:22:25,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000627814 s, 833 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:22:25,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000753686 s, 492 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:25,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:25,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091186 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:27,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:28,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916257 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:22:29,746::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:22:29,747::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:22:30,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:30,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912671 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:32,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:32,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912153 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:34,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:34,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541597 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:22:35,449::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:22:35,449::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:22:35,474::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682206 s, 544 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:22:35,477::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000719984 s, 726 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:36,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:36,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913952 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:22:38,287::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:22:38,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:39,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911223 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:22:39,756::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:22:39,757::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:22:41,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:41,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913713 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:43,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:43,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542372 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:22:45,478::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:22:45,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065801 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:22:45,492::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:22:45,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666299 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:45,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:45,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0996007 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:48,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:48,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0932973 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:22:49,768::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:22:49,769::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:22:50,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:50,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09194 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:52,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:52,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091374 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:22:53,299::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:22:54,275::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:22:54,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:54,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912373 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:22:55,506::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:22:55,517::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 16:22:55,520::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670984 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:22:55,538::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:22:55,539::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:22:55,542::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:22:55,544::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:22:55,544::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:22:55,544::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:22:55,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:22:55,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653634 s, 800 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:56,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:56,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912448 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:22:58,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:22:58,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913042 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:22:59,780::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:22:59,781::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:23:00,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:01,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555047 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:03,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:03,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911187 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:05,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:23:05,565::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:23:05,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:23:05,588::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647267 s, 808 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:23:05,591::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679983 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:05,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.130435 s, 7.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:07,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:07,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912449 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:23:08,302::task::595::Storage.TaskManager.Task::(_updateState) Task=`9245002d-a518-4921-b2d5-f3a5374cd27d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:23:08,302::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:23:08,305::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:23:08,305::task::1191::Storage.TaskManager.Task::(prepare) Task=`9245002d-a518-4921-b2d5-f3a5374cd27d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:23:08,305::task::595::Storage.TaskManager.Task::(_updateState) Task=`9245002d-a518-4921-b2d5-f3a5374cd27d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:23:08,305::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:23:08,306::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:23:08,306::task::993::Storage.TaskManager.Task::(_decref) Task=`9245002d-a518-4921-b2d5-f3a5374cd27d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:23:08,333::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:23:09,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:23:09,793::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:23:09,794::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:23:09,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913811 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:11,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:11,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913864 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:13,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:14,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.114895 s, 8.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:23:15,606::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:23:15,607::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:23:15,632::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000592839 s, 626 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:23:15,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660996 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:16,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:16,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0997063 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:18,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:18,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543662 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:23:19,800::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:23:19,801::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:23:20,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:21,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911578 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:23,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:23,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.093197 s, 11.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:23:23,345::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:23:24,308::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:23:25,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:25,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912656 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:23:25,640::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 16:23:25,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:23:25,667::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636912 s, 821 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:23:25,672::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:23:25,674::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:23:25,677::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:23:25,679::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:23:25,679::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:23:25,679::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:23:25,680::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:23:25,692::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659652 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:27,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:27,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912394 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:29,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:29,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912728 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:23:29,812::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:23:29,813::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:23:31,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:31,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913451 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:33,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:33,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911366 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:23:35,677::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:23:35,689::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692531 s, 755 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:23:35,695::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:23:35,706::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630842 s, 588 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:35,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:35,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091318 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:37,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:23:38,357::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:23:38,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518065 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:23:39,821::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:23:39,822::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:23:40,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:40,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913308 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:42,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:43,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540913 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:45,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:45,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913693 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:23:45,695::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:23:45,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0023587 s, 222 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:23:45,716::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:23:45,731::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000719217 s, 516 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:47,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:47,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091376 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:49,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:49,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911896 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:23:49,831::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:23:49,832::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:23:51,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:51,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912047 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:23:53,369::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:23:53,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:53,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913407 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:23:54,349::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 16:23:55,733::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:23:55,744::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:23:55,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660296 s, 792 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:23:55,762::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00129096 s, 287 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:55,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:55,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914327 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:23:57,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:23:58,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548578 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:23:59,841::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:23:59,842::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:24:00,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:00,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912859 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:02,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:02,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0943947 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:04,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:04,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912953 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:24:05,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:24:05,778::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:24:05,793::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678747 s, 771 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:24:05,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635571 s, 584 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:06,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:07,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544204 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:24:08,371::task::595::Storage.TaskManager.Task::(_updateState) Task=`613cc067-ae68-4669-9f32-c8e81d7d2ed2`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:24:08,372::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:24:08,375::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:24:08,375::task::1191::Storage.TaskManager.Task::(prepare) Task=`613cc067-ae68-4669-9f32-c8e81d7d2ed2`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:24:08,375::task::595::Storage.TaskManager.Task::(_updateState) Task=`613cc067-ae68-4669-9f32-c8e81d7d2ed2`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:24:08,375::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:24:08,376::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:24:08,376::task::993::Storage.TaskManager.Task::(_decref) Task=`613cc067-ae68-4669-9f32-c8e81d7d2ed2`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:24:08,402::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:24:09,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:09,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091304 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:24:09,854::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:24:09,855::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:24:11,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:11,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920365 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:13,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:13,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912469 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:15,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:24:15,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:24:15,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:24:15,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000605643 s, 613 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:24:15,908::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000633225 s, 826 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:15,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.142005 s, 7.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:18,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:18,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912338 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:24:19,861::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:24:19,862::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:24:20,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:20,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913552 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:22,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:22,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.520679 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:24:23,414::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:24:24,381::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:24:24,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:24,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913989 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:24:25,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:24:25,922::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:24:25,922::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000721392 s, 514 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:24:25,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000618343 s, 846 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:26,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:27,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911242 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:29,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:29,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912415 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:24:29,872::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:24:29,873::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:24:31,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:31,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920048 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:33,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:33,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546725 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:35,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:24:35,969::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:24:35,980::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:24:35,992::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00086512 s, 429 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:24:35,993::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000714203 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:36,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.123992 s, 8.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:38,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:38,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912864 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:24:38,425::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:24:39,884::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:24:39,886::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:24:40,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:40,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519985 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:42,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:42,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913246 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:44,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:44,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913562 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:24:45,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:24:46,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000713842 s, 520 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:24:46,012::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:24:46,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679688 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:46,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:47,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912452 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:49,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:49,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091502 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:24:49,893::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:24:49,894::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:24:51,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:51,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548156 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:24:53,437::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:24:53,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:54,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504768 s, 2.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:24:54,414::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 16:24:56,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:24:56,041::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000776074 s, 478 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:24:56,041::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:24:56,058::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000734447 s, 712 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:56,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:56,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913638 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:24:58,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:24:58,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542079 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:24:59,905::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:24:59,906::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:25:00,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:01,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914929 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:03,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:03,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912692 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:05,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:05,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912947 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:25:06,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:25:06,072::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:25:06,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665105 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:25:06,085::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000848079 s, 617 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:07,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:07,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912062 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:25:08,448::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d1a8a12-a15c-4257-94ae-ca03eca0d77d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:25:08,449::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:25:08,452::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:25:08,453::task::1191::Storage.TaskManager.Task::(prepare) Task=`5d1a8a12-a15c-4257-94ae-ca03eca0d77d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:25:08,453::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d1a8a12-a15c-4257-94ae-ca03eca0d77d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:25:08,453::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:25:08,453::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:25:08,453::task::993::Storage.TaskManager.Task::(_decref) Task=`5d1a8a12-a15c-4257-94ae-ca03eca0d77d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:25:08,479::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:25:09,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:09,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912638 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:25:09,917::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:25:09,918::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:25:11,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:12,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54156 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:14,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:14,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914752 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:25:16,092::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:25:16,092::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:25:16,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627213 s, 592 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:25:16,117::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00231576 s, 226 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:16,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:16,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0987512 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:18,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:18,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912695 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:25:19,924::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:25:19,925::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:25:20,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:20,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913019 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:22,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:22,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912509 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:25:23,491::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:25:24,445::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:25:24,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:24,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912404 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:25:26,119::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:25:26,132::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000760832 s, 488 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:25:26,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:25:26,147::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675788 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:26,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:27,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917677 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:29,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:29,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918235 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:25:29,937::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:25:29,938::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:25:31,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:31,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548489 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:33,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:33,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917979 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:35,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:35,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912795 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:25:36,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:25:36,160::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:25:36,171::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0049451 s, 75.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:25:36,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000625396 s, 836 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:37,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:38,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912647 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:25:38,502::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:25:39,949::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:25:39,950::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:25:40,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:40,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915611 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:42,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:42,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547293 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:44,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:44,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913236 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:25:46,180::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:25:46,181::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:25:46,207::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000751488 s, 494 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:25:46,209::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000763852 s, 685 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:46,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:47,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917101 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:49,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:49,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916283 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:25:49,960::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:25:49,961::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:25:51,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:51,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915476 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:53,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:25:53,515::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:25:53,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557414 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:25:54,478::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:25:55,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:55,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913851 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:25:56,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:25:56,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000603532 s, 615 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:25:56,231::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:25:56,243::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720998 s, 725 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:25:57,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:25:58,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914272 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:25:59,972::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:25:59,973::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:26:00,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:00,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091354 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:02,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:02,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915191 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:04,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:04,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914337 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:26:06,244::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:26:06,256::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:26:06,271::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647854 s, 573 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:26:06,274::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668864 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:06,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:06,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.508744 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:26:08,518::task::595::Storage.TaskManager.Task::(_updateState) Task=`35a5a31e-1e16-4d0e-9dea-0ff56b5b0491`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:26:08,518::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:26:08,521::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:26:08,521::task::1191::Storage.TaskManager.Task::(prepare) Task=`35a5a31e-1e16-4d0e-9dea-0ff56b5b0491`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:26:08,521::task::595::Storage.TaskManager.Task::(_updateState) Task=`35a5a31e-1e16-4d0e-9dea-0ff56b5b0491`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:26:08,521::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:26:08,521::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:26:08,522::task::993::Storage.TaskManager.Task::(_decref) Task=`35a5a31e-1e16-4d0e-9dea-0ff56b5b0491`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:26:08,549::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:26:08,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:09,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912931 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:26:09,984::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:26:09,985::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:26:11,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:11,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912877 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:13,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:13,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914557 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:15,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:15,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100026 s, 10.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:26:16,277::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:26:16,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:26:16,301::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661465 s, 561 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:26:16,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000792511 s, 660 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:17,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:17,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912617 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:19,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:26:19,990::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:26:19,992::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:26:20,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914178 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:22,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:22,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912507 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:26:23,561::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:26:24,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:24,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914469 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:26:24,511::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:26:26,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:26:26,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:26:26,381::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636019 s, 822 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:26:26,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:26:26,398::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00123086 s, 301 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:26,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.12749 s, 8.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:28,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:28,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914341 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:26:30,002::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:26:30,003::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:26:30,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:30,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091387 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:32,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:32,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091388 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:34,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:34,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964276 s, 10.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:26:36,400::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:26:36,413::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:26:36,428::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659052 s, 794 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:26:36,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680891 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:36,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:36,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912246 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:26:38,574::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:26:39,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:39,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913427 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:26:40,012::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:26:40,013::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:26:41,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:41,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913571 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:43,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:43,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.521137 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:45,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:45,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100109 s, 10.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:26:46,438::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:26:46,448::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:26:46,462::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000644277 s, 812 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:26:46,466::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000611983 s, 606 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:48,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:48,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912637 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:26:50,024::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:26:50,025::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:26:50,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:50,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911667 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:52,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:52,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549165 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:26:53,586::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:26:54,545::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:26:54,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:55,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511563 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:26:56,476::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:26:56,486::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:26:56,497::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642133 s, 578 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:26:56,499::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000950416 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:57,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:26:57,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091935 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:26:59,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:27:00,035::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:27:00,036::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:27:00,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54965 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:02,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:02,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913609 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:04,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:04,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914505 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:06,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:27:06,874::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:27:06,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:27:06,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680793 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:27:06,906::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000763455 s, 685 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:06,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5848 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:27:08,589::task::595::Storage.TaskManager.Task::(_updateState) Task=`6640f217-08d4-40ca-bead-197f3d7b81e6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:27:08,590::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:27:08,592::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:27:08,593::task::1191::Storage.TaskManager.Task::(prepare) Task=`6640f217-08d4-40ca-bead-197f3d7b81e6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:27:08,593::task::595::Storage.TaskManager.Task::(_updateState) Task=`6640f217-08d4-40ca-bead-197f3d7b81e6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:27:08,593::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:27:08,593::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:27:08,593::task::993::Storage.TaskManager.Task::(_decref) Task=`6640f217-08d4-40ca-bead-197f3d7b81e6`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:27:08,621::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:27:08,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:09,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913472 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:27:10,047::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:27:10,048::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:27:11,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:11,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503529 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:13,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:14,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.512204 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:16,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:16,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100529 s, 10.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:27:16,912::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:27:16,924::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:27:16,935::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00517057 s, 71.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:27:16,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661721 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:18,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:18,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914537 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:27:20,057::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:27:20,058::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:27:20,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:20,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925988 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:22,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:23,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549172 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:27:23,640::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:27:24,578::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:27:25,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:25,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912738 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:27:26,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:27:26,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:27:26,968::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643616 s, 576 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:27:26,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000603995 s, 866 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:27,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:27,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913917 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:29,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:29,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542977 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:27:30,070::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:27:30,071::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:27:31,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:32,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913303 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:34,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:34,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911533 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:36,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:36,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091142 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:27:36,975::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:27:36,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000717527 s, 517 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:27:36,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:27:37,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000634202 s, 825 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:38,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:38,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912045 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:27:38,652::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:27:40,078::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:27:40,080::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:27:40,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:40,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912479 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:42,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:42,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914248 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:44,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:44,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914833 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:46,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:46,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910985 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:27:47,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:27:47,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:27:47,024::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674047 s, 550 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:27:47,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677406 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:48,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:49,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518809 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:27:50,091::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:27:50,093::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:27:51,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:51,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091079 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:53,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:27:53,664::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:27:53,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911805 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:27:54,611::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:27:55,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:55,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911448 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:27:57,028::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 16:27:57,036::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:27:57,051::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000698321 s, 531 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:27:57,060::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:27:57,062::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:27:57,065::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:27:57,066::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:27:57,066::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:27:57,066::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:27:57,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:27:57,079::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669354 s, 781 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:57,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:27:57,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913851 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:27:59,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:00,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0976 s, 10.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:28:00,099::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:28:00,100::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:28:02,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:02,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912741 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:04,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:04,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913747 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:06,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:06,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914359 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:28:07,058::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:28:07,070::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704369 s, 527 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:28:07,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:28:07,094::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632828 s, 826 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:08,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:08,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911864 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:28:08,667::task::595::Storage.TaskManager.Task::(_updateState) Task=`0119611b-feb1-424d-a4f0-81eea5eac336`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:28:08,667::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:28:08,670::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:28:08,670::task::1191::Storage.TaskManager.Task::(prepare) Task=`0119611b-feb1-424d-a4f0-81eea5eac336`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:28:08,670::task::595::Storage.TaskManager.Task::(_updateState) Task=`0119611b-feb1-424d-a4f0-81eea5eac336`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:28:08,671::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:28:08,671::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:28:08,671::task::993::Storage.TaskManager.Task::(_decref) Task=`0119611b-feb1-424d-a4f0-81eea5eac336`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:28:08,696::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:28:10,108::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:28:10,109::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:28:10,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:11,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518895 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:13,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:13,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914447 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:15,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:15,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.107938 s, 9.5 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:28:17,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:28:17,088::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657499 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:28:17,097::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:28:17,109::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678994 s, 770 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:17,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:17,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911831 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:19,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:19,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913038 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:28:20,116::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:28:20,117::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:28:21,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:22,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913471 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:28:23,707::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:28:24,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:24,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.133238 s, 7.7 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:28:24,643::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:28:26,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:26,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911906 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:28:27,090::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:28:27,102::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:28:27,103::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:28:27,106::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:28:27,108::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:28:27,108::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:28:27,108::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:28:27,109::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:28:27,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:28:27,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00081461 s, 455 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:28:27,135::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666495 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:28,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:28,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913346 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:28:30,125::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:28:30,126::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:28:30,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:30,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091222 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:32,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:32,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913691 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:34,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:35,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544924 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:28:37,136::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:28:37,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:28:37,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000781236 s, 475 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:28:37,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676795 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:37,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:37,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916874 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:28:38,719::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:28:39,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:39,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910498 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:28:40,137::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:28:40,139::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:28:41,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:41,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912184 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:43,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:43,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911872 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:45,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:45,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100118 s, 10.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:28:47,169::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:28:47,180::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:28:47,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000586334 s, 633 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:28:47,197::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00111811 s, 468 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:47,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:48,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:50,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:28:50,150::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:28:50,152::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:28:50,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917646 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:52,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:52,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913149 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:28:53,730::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:28:54,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:54,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911997 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:28:54,676::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:28:56,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:56,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515384 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:28:57,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:28:57,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063349 s, 586 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:28:57,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:28:57,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000783813 s, 667 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:28:58,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:28:59,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914896 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:29:00,156::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:29:00,157::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:29:01,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:01,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091262 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:03,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:03,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091286 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:05,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:05,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911613 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:29:07,227::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:29:07,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:29:07,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632407 s, 587 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:29:07,253::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0011344 s, 461 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:07,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:07,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091111 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:29:08,733::task::595::Storage.TaskManager.Task::(_updateState) Task=`03b099b9-adae-4753-8737-2c9131fc71e7`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:29:08,733::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:29:08,736::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:29:08,736::task::1191::Storage.TaskManager.Task::(prepare) Task=`03b099b9-adae-4753-8737-2c9131fc71e7`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:29:08,736::task::595::Storage.TaskManager.Task::(_updateState) Task=`03b099b9-adae-4753-8737-2c9131fc71e7`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:29:08,736::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:29:08,736::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:29:08,737::task::993::Storage.TaskManager.Task::(_decref) Task=`03b099b9-adae-4753-8737-2c9131fc71e7`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:29:08,763::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:29:09,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:09,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914424 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:29:10,168::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:29:10,169::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:29:11,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:11,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912443 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:13,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:13,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913666 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:15,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:16,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915896 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:29:17,256::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:29:17,268::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:29:17,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00430433 s, 86.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:29:17,281::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000991878 s, 527 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:18,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:18,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091172 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:29:20,180::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:29:20,181::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:29:20,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:20,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.513711 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:22,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:22,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091368 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:29:23,774::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:29:24,713::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:29:24,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:25,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913058 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:27,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:29:27,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:29:27,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:29:27,599::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651129 s, 803 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:29:27,601::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662166 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:27,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.57909 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:29,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:29,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912626 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:29:30,192::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:29:30,194::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:29:31,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:31,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913683 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:33,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:33,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911596 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:35,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:36,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913297 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:29:37,609::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:29:37,621::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:29:37,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00408248 s, 128 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:29:37,635::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000803774 s, 462 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:38,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:38,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911827 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:29:38,786::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:29:40,200::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:29:40,201::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:29:40,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:40,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912997 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:42,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:42,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914162 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:44,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:44,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913598 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:46,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:46,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914368 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:29:47,640::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:29:47,651::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:29:47,663::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00474261 s, 78.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:29:47,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000612278 s, 854 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:48,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:48,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912391 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:29:50,212::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:29:50,214::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:29:50,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:51,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919493 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:53,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:53,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914411 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:29:53,798::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:29:54,746::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:29:55,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:55,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912335 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:57,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:57,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132424 s, 7.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:29:57,669::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:29:57,680::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:29:57,691::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066663 s, 557 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:29:57,693::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000623171 s, 839 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:29:59,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:29:59,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914433 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:30:00,223::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:30:00,224::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:30:01,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:01,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916293 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:03,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:03,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915053 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:05,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:06,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.506097 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:30:07,698::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:30:07,710::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:30:07,721::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657526 s, 564 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:30:07,724::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674657 s, 775 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:08,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:08,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912555 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:30:08,801::task::595::Storage.TaskManager.Task::(_updateState) Task=`7803369a-6ca6-4111-8188-73605faa4673`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:30:08,801::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:30:08,804::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:30:08,804::task::1191::Storage.TaskManager.Task::(prepare) Task=`7803369a-6ca6-4111-8188-73605faa4673`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:30:08,804::task::595::Storage.TaskManager.Task::(_updateState) Task=`7803369a-6ca6-4111-8188-73605faa4673`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:30:08,805::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:30:08,805::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:30:08,805::task::993::Storage.TaskManager.Task::(_decref) Task=`7803369a-6ca6-4111-8188-73605faa4673`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:30:08,830::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:30:10,231::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:30:10,232::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:30:10,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:11,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515802 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:13,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:13,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0937978 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:15,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:15,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.565032 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:30:17,726::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:30:17,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:30:17,749::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00440142 s, 84.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:30:17,751::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630383 s, 830 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:17,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:17,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913356 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:19,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:20,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921506 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:30:20,240::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:30:20,241::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:30:22,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:22,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914649 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:30:23,842::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:30:24,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:24,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911539 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:30:24,781::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:30:26,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:26,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0962042 s, 10.6 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:30:27,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:30:27,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:30:27,777::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000737257 s, 503 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:30:27,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700251 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:28,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:28,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913876 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:30:30,251::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:30:30,252::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:30:30,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:30,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913428 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:32,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:32,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913001 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:34,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:34,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912816 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:36,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:37,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913892 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:30:37,782::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:30:37,793::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:30:37,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689259 s, 538 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:30:37,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650795 s, 804 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:30:38,853::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:30:39,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:39,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913094 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:30:40,263::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:30:40,265::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:30:41,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:41,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913673 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:43,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:43,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911729 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:45,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:45,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913672 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:47,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:47,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912831 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:30:47,810::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:30:47,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:30:47,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00572239 s, 64.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:30:47,833::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667294 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:49,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:49,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913241 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:30:50,275::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:30:50,276::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:30:51,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:51,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913417 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:30:53,865::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:30:53,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:53,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912083 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:30:54,811::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:30:55,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:56,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914365 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:30:57,837::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:30:57,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:30:57,859::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000732607 s, 506 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:30:57,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662173 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:30:58,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:30:58,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911618 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:00,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:31:00,284::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:31:00,285::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:31:00,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920228 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:02,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:02,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557951 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:04,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:05,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912501 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:07,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:07,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091277 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:31:07,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:31:07,877::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:31:07,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675779 s, 549 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:31:07,889::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000644353 s, 812 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:31:08,868::task::595::Storage.TaskManager.Task::(_updateState) Task=`63cfaf1e-4a38-4185-9a6f-9c5cc3c809f9`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:31:08,868::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:31:08,871::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:31:08,871::task::1191::Storage.TaskManager.Task::(prepare) Task=`63cfaf1e-4a38-4185-9a6f-9c5cc3c809f9`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:31:08,871::task::595::Storage.TaskManager.Task::(_updateState) Task=`63cfaf1e-4a38-4185-9a6f-9c5cc3c809f9`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:31:08,872::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:31:08,872::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:31:08,872::task::993::Storage.TaskManager.Task::(_decref) Task=`63cfaf1e-4a38-4185-9a6f-9c5cc3c809f9`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:31:08,897::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:31:09,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:09,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510304 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:31:10,296::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:31:10,297::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:31:11,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:11,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091191 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:13,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:13,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913461 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:15,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:16,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921429 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:31:17,893::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:31:17,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:31:17,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00320872 s, 116 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:31:17,917::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000736856 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:18,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:18,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917955 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:20,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:31:20,304::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:31:20,305::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:31:20,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920007 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:22,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:22,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545348 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:31:23,909::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:31:24,841::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:31:24,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:25,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912583 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:27,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:27,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915968 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:31:27,921::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:31:27,933::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:31:27,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000781254 s, 475 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:31:27,951::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688095 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:29,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:29,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132343 s, 7.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:31:30,313::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:31:30,314::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:31:31,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:31,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913089 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:33,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:33,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912223 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:35,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:35,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913705 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:37,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:37,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911942 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:31:37,954::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:31:37,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000743797 s, 499 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:31:37,968::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:31:37,980::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660137 s, 792 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:31:38,921::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:31:39,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:39,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912707 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:31:40,325::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:31:40,326::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:31:41,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:42,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912304 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:44,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:44,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:46,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:46,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968267 s, 10.6 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:31:47,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:31:47,993::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617423 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:31:47,995::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:31:48,008::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000701659 s, 745 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:48,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:48,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913989 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:31:50,337::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:31:50,338::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:31:50,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:50,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911756 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:52,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:52,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913302 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:31:53,933::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:31:54,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:31:54,872::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:31:54,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091319 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:56,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:57,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912168 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:31:58,010::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:31:58,022::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:31:58,023::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000613312 s, 605 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:31:58,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000738426 s, 708 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:31:59,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:31:59,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915135 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:32:00,349::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:32:00,351::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:32:01,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:01,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914026 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:03,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:03,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914206 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:05,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:05,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918901 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:07,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:08,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502115 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:32:08,038::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:32:08,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:32:08,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067032 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:32:08,062::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064595 s, 810 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:32:08,936::task::595::Storage.TaskManager.Task::(_updateState) Task=`98b8b9b3-6616-4f00-bc7b-b9567a633af6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:32:08,936::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:32:08,939::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:32:08,939::task::1191::Storage.TaskManager.Task::(prepare) Task=`98b8b9b3-6616-4f00-bc7b-b9567a633af6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:32:08,939::task::595::Storage.TaskManager.Task::(_updateState) Task=`98b8b9b3-6616-4f00-bc7b-b9567a633af6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:32:08,939::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:32:08,939::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:32:08,940::task::993::Storage.TaskManager.Task::(_decref) Task=`98b8b9b3-6616-4f00-bc7b-b9567a633af6`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:32:08,966::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:32:10,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:32:10,362::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:32:10,363::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:32:10,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557906 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:12,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:12,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912763 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:14,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:14,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913308 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:16,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:16,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0988719 s, 10.4 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:32:18,067::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:32:18,078::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:32:18,089::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00476968 s, 77.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:32:18,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660551 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:19,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:19,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913796 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:32:20,370::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:32:20,371::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:32:21,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:21,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913768 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:23,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:23,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545452 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:32:23,981::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:32:24,903::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:32:25,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:26,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542407 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:32:28,095::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:32:28,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:32:28,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00540071 s, 68.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:32:28,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683589 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:28,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:28,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550298 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:32:30,382::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:32:30,383::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:32:31,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:31,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541652 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:33,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:34,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504485 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:36,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:36,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913257 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:32:38,124::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:32:38,135::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:32:38,147::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000711514 s, 521 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:32:38,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000746519 s, 701 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:38,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:38,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913821 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:32:38,994::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:32:40,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:32:40,395::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:32:40,396::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:32:40,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09147 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:42,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:42,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914009 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:44,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:44,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915033 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:46,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:46,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091393 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:32:48,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:32:48,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:32:48,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664991 s, 558 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:32:48,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632416 s, 827 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:48,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:48,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.112525 s, 9.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:32:50,402::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:32:50,403::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:32:50,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:51,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091346 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:53,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:53,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914284 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:32:54,006::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:32:54,936::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:32:55,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:55,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913949 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:57,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:57,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912037 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:32:58,180::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 16:32:58,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:32:58,214::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-77::DEBUG::2014-07-14 16:32:58,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680732 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:32:58,215::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:32:58,219::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:32:58,220::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:32:58,220::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:32:58,221::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:32:58,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:32:58,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672611 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:32:59,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:32:59,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912894 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:33:00,410::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:33:00,411::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:33:01,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:01,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912206 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:03,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:03,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913325 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:05,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:05,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912539 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:07,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:07,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913695 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:33:08,219::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:33:08,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618395 s, 600 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:33:08,237::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:33:08,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000618912 s, 845 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:33:09,009::task::595::Storage.TaskManager.Task::(_updateState) Task=`52aeaebc-f4aa-4b03-b53b-dbe93b778621`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:33:09,009::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:33:09,012::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:33:09,012::task::1191::Storage.TaskManager.Task::(prepare) Task=`52aeaebc-f4aa-4b03-b53b-dbe93b778621`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:33:09,013::task::595::Storage.TaskManager.Task::(_updateState) Task=`52aeaebc-f4aa-4b03-b53b-dbe93b778621`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:33:09,013::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:33:09,013::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:33:09,013::task::993::Storage.TaskManager.Task::(_decref) Task=`52aeaebc-f4aa-4b03-b53b-dbe93b778621`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:33:09,038::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:33:10,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:33:10,423::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:33:10,424::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:33:10,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557474 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:12,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:13,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38881 s, 737 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:16,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:16,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926334 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:18,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:33:18,240::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:33:18,780::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:33:18,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629941 s, 589 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:33:18,796::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713536 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:18,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.567004 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:33:20,430::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:33:20,432::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:33:20,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:21,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557889 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:23,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:23,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914083 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:33:24,050::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:33:24,974::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:33:25,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:26,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.505634 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:28,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:28,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546054 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:33:28,800::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 16:33:29,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:33:29,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000609303 s, 858 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:33:29,141::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:33:29,142::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:33:29,145::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:33:29,147::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:33:29,147::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:33:29,147::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:33:29,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:33:29,160::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676188 s, 549 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:33:30,443::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:33:30,444::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:33:30,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:31,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542411 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:33,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:33,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913579 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:35,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:35,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912966 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:37,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:37,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914754 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:33:39,062::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 16:33:39,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:33:39,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00073051 s, 716 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:33:39,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:33:39,175::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671295 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:39,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:39,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09135 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:33:40,455::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:33:40,456::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:33:41,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:41,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913871 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:43,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:43,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913261 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:45,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:46,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097911 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:48,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:48,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913465 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:33:49,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:33:49,167::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000721965 s, 724 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:33:49,178::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:33:49,189::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687048 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:50,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:50,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911961 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:33:50,468::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:33:50,469::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:33:52,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:52,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913245 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:33:54,074::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:33:54,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:54,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911773 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:33:55,006::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:33:56,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:56,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915282 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:33:58,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:33:58,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913258 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:33:59,171::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:33:59,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665301 s, 786 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:33:59,192::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:33:59,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000554993 s, 668 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:34:00,475::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:34:00,476::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:34:00,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:00,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914964 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:02,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:03,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545702 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:05,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:06,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511784 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:08,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:08,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913473 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:34:09,078::task::595::Storage.TaskManager.Task::(_updateState) Task=`14057666-3210-478b-8957-5c64ccb6e23b`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:34:09,078::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:34:09,081::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:34:09,081::task::1191::Storage.TaskManager.Task::(prepare) Task=`14057666-3210-478b-8957-5c64ccb6e23b`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:34:09,081::task::595::Storage.TaskManager.Task::(_updateState) Task=`14057666-3210-478b-8957-5c64ccb6e23b`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:34:09,081::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:34:09,081::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:34:09,082::task::993::Storage.TaskManager.Task::(_decref) Task=`14057666-3210-478b-8957-5c64ccb6e23b`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:34:09,109::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 16:34:09,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:34:09,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643239 s, 813 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:34:09,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:34:09,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620374 s, 598 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:10,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:10,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913602 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:34:10,484::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:34:10,485::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:34:12,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:12,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911466 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:14,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:14,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914766 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:16,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:16,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911413 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:18,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:18,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910971 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:34:19,202::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:34:19,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703695 s, 743 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:34:19,221::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:34:19,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650874 s, 570 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:34:20,494::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:34:20,495::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:34:20,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:20,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091273 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:22,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:23,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091419 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:34:24,122::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:34:25,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:34:25,041::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:34:25,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913315 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:27,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:27,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913018 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:34:29,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:34:29,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661555 s, 791 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:34:29,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:34:29,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000745662 s, 498 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:29,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:29,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540995 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:34:30,506::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:34:30,508::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:34:31,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:31,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913905 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:33,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:34,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912114 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:36,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:36,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913781 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:38,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:38,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912506 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:34:39,134::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 16:34:39,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:34:39,246::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678278 s, 771 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:34:39,250::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:34:39,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000710393 s, 522 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:40,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:34:40,516::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:34:40,517::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:34:40,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.501128 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:42,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:44,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37103 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:46,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:46,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0965942 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:48,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:48,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923664 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:34:49,250::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:34:49,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685763 s, 763 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:34:49,265::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:34:49,277::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668991 s, 555 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:34:50,524::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:34:50,525::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:34:50,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:50,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913377 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:52,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:52,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091311 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:34:54,146::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:34:54,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:54,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091416 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:34:55,074::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:34:56,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:34:57,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913594 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:59,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:34:59,606::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:34:59,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:34:59,619::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000737554 s, 709 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:34:59,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657421 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:34:59,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542502 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:35:00,536::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:35:00,537::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:35:01,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:01,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912621 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:03,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:04,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544476 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:06,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:06,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:08,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:09,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543039 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:35:09,149::task::595::Storage.TaskManager.Task::(_updateState) Task=`c99ce855-2b5a-4690-9118-ff788690e862`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:35:09,150::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:35:09,152::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:35:09,153::task::1191::Storage.TaskManager.Task::(prepare) Task=`c99ce855-2b5a-4690-9118-ff788690e862`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:35:09,153::task::595::Storage.TaskManager.Task::(_updateState) Task=`c99ce855-2b5a-4690-9118-ff788690e862`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:35:09,153::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:35:09,153::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:35:09,153::task::993::Storage.TaskManager.Task::(_decref) Task=`c99ce855-2b5a-4690-9118-ff788690e862`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:35:09,180::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 16:35:09,641::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:35:09,653::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:35:09,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000742586 s, 704 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:35:09,668::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000754167 s, 492 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:35:10,543::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:35:10,545::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:35:11,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:11,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540722 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:13,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:13,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912051 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:15,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:15,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108797 s, 9.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:17,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:18,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912483 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:35:19,673::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:35:19,683::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:35:19,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0029725 s, 125 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:35:19,696::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648726 s, 806 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:20,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:20,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912003 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:35:20,556::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:35:20,557::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:35:22,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:22,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913051 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:35:24,193::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:35:24,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:24,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912002 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:35:25,106::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:35:26,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:26,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091394 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:28,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:28,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912292 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:35:29,700::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:35:29,711::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:35:29,723::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617128 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:35:29,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658175 s, 795 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:35:30,568::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:35:30,569::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:35:30,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:30,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913245 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:32,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:32,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913704 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:34,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:35,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913028 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:37,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:37,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920029 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:39,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:35:39,207::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:35:39,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913462 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:35:39,729::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:35:39,741::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:35:39,742::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000735818 s, 504 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:35:39,753::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000613074 s, 853 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:35:40,580::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:35:40,582::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:35:41,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:41,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913694 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:43,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:43,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912414 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:45,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:45,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106192 s, 9.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:47,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:47,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911656 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:35:49,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:35:49,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:35:49,778::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000614485 s, 604 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:35:49,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000736579 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:49,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:49,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912636 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:35:50,593::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:35:50,594::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:35:51,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:52,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915511 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:54,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:54,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911487 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:35:54,223::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:35:55,138::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:35:56,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:56,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912941 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:35:58,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:35:58,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912564 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:35:59,782::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:35:59,794::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:35:59,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000767803 s, 483 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:35:59,807::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681502 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:00,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:00,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913382 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:36:00,604::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:36:00,605::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:36:02,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:02,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912927 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:04,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:04,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912707 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:06,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:07,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519052 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:36:09,226::task::595::Storage.TaskManager.Task::(_updateState) Task=`2507b98a-4633-4b4f-9564-2792c720e1e6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:36:09,226::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:36:09,229::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:36:09,229::task::1191::Storage.TaskManager.Task::(prepare) Task=`2507b98a-4633-4b4f-9564-2792c720e1e6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:36:09,229::task::595::Storage.TaskManager.Task::(_updateState) Task=`2507b98a-4633-4b4f-9564-2792c720e1e6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:36:09,230::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:36:09,230::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:36:09,230::task::993::Storage.TaskManager.Task::(_decref) Task=`2507b98a-4633-4b4f-9564-2792c720e1e6`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:36:09,257::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:36:09,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:09,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912075 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:36:09,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:36:09,823::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:36:09,838::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000712799 s, 520 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:36:09,841::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694465 s, 753 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:36:10,614::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:36:10,615::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:36:11,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:11,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914186 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:13,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:13,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922283 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:15,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:15,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914367 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:17,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:19,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39974 s, 732 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:36:19,842::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:36:19,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:36:19,865::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00580729 s, 63.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:36:19,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629526 s, 831 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:36:20,624::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:36:20,625::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:36:21,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:21,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541092 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:23,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:36:24,269::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:36:24,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.506057 s, 2.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:36:25,170::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:36:26,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:26,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912289 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:28,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:28,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911991 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:36:29,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:36:29,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:36:29,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00453299 s, 81.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:36:29,895::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694317 s, 753 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:36:30,635::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:36:30,636::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:36:30,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:30,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913149 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:32,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:32,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913136 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:34,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:35,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913245 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:37,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:37,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914518 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:39,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:36:39,282::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:36:39,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544402 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:36:39,900::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:36:39,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:36:39,928::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649376 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:36:39,929::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671691 s, 779 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:36:40,649::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:36:40,654::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:36:41,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:42,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504042 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:44,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:44,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913284 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:46,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:46,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999767 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:48,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:48,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0980019 s, 10.4 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:36:49,944::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:36:49,955::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:36:49,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000745744 s, 497 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:36:49,969::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640367 s, 817 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:36:50,660::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:36:50,661::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:36:50,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:50,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912676 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:52,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:52,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915476 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:36:54,294::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:36:54,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:55,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912429 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:36:55,202::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:36:57,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:57,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914895 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:36:59,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:36:59,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503654 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:36:59,971::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:36:59,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000786721 s, 472 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:36:59,984::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:36:59,997::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00073355 s, 713 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:37:00,672::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:37:00,673::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:37:01,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:01,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912587 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:03,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:03,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912939 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:05,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:06,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54345 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:08,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:08,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914987 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:37:09,297::task::595::Storage.TaskManager.Task::(_updateState) Task=`d209b1e2-40fd-4723-ba75-278645c2f47c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:37:09,297::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:37:09,300::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:37:09,300::task::1191::Storage.TaskManager.Task::(prepare) Task=`d209b1e2-40fd-4723-ba75-278645c2f47c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:37:09,300::task::595::Storage.TaskManager.Task::(_updateState) Task=`d209b1e2-40fd-4723-ba75-278645c2f47c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:37:09,300::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:37:09,301::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:37:09,301::task::993::Storage.TaskManager.Task::(_decref) Task=`d209b1e2-40fd-4723-ba75-278645c2f47c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:37:09,328::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:37:09,998::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:37:10,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:37:10,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000616458 s, 602 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:37:10,024::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675604 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:10,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:37:10,683::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:37:10,685::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:37:10,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913396 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:12,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:12,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914516 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:14,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:14,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914715 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:16,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:17,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913213 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:19,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:19,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913739 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:37:20,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:37:20,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:37:20,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00536805 s, 69.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:37:20,052::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000797149 s, 656 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:37:20,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:37:20,696::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:37:21,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:21,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913584 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:23,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:23,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502677 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:37:24,345::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:37:25,236::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:37:25,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:25,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911529 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:27,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:28,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914231 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:37:30,053::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:37:30,065::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:37:30,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00584538 s, 63.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:37:30,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000868244 s, 602 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:30,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:30,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913195 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:37:30,703::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:37:30,704::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:37:32,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:32,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913106 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:34,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:34,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920741 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:36,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:36,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.115942 s, 8.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:38,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:38,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911594 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:37:39,358::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:37:40,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:37:40,092::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:37:40,103::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00060271 s, 616 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:37:40,105::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640912 s, 816 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:40,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:37:40,714::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:37:40,715::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:37:40,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912304 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:42,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:43,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542275 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:45,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:45,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539889 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:48,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:48,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912877 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:37:50,109::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:37:50,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:37:50,139::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00302468 s, 123 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:37:50,139::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00128952 s, 406 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:50,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:50,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917292 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:37:50,727::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:37:50,728::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:37:52,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:52,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913441 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:37:54,370::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:37:54,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:55,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557719 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:37:55,269::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:37:57,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:37:57,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547788 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:37:59,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:38:00,144::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Dummy-49::DEBUG::2014-07-14 16:38:00,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545007 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:38:00,161::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:38:00,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628789 s, 590 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:38:00,188::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:38:00,191::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:38:00,197::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:38:00,200::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:38:00,201::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:38:00,202::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:38:00,204::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:38:00,221::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00454167 s, 115 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:38:00,735::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:38:00,737::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:38:02,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:02,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913074 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:04,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:04,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912222 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:06,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:06,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913461 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:08,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:08,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091452 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:38:09,372::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a2b1abd-78d8-4582-b3b5-352f481bedcf`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:38:09,373::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:38:09,376::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:38:09,376::task::1191::Storage.TaskManager.Task::(prepare) Task=`5a2b1abd-78d8-4582-b3b5-352f481bedcf`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:38:09,376::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a2b1abd-78d8-4582-b3b5-352f481bedcf`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:38:09,377::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:38:09,377::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:38:09,377::task::993::Storage.TaskManager.Task::(_decref) Task=`5a2b1abd-78d8-4582-b3b5-352f481bedcf`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:38:09,404::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:38:10,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:38:10,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630672 s, 588 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:38:10,227::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:38:10,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636701 s, 821 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:10,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:38:10,751::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:38:10,752::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:38:10,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918667 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:12,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:12,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912241 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:14,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:15,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914987 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:17,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:17,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911973 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:19,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:19,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915911 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:38:20,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:38:20,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684761 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:38:20,242::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:38:20,253::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671892 s, 778 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:38:20,764::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:38:20,765::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:38:21,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:21,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540697 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:23,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:23,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914482 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:38:24,417::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:38:25,306::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:38:25,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:26,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912728 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:28,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:28,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914592 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:30,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:38:30,222::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-29::DEBUG::2014-07-14 16:38:30,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:38:30,333::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667378 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:30,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.129579 s, 7.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:38:30,371::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:38:30,373::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:38:30,377::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:38:30,380::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:38:30,381::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:38:30,381::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:38:30,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:38:30,400::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00344014 s, 108 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:38:30,776::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:38:30,777::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:38:32,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:32,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914213 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:34,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:34,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915538 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:36,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:37,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544161 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:39,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:38:39,429::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:38:40,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54519 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:38:40,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:38:40,360::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699448 s, 748 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:38:40,405::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:38:40,417::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668552 s, 555 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:38:40,788::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:38:40,789::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:38:42,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:42,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915182 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:44,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:44,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540417 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:46,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:47,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914977 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:49,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:49,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914288 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:38:50,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:38:50,375::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689102 s, 759 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:38:50,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:38:50,431::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675532 s, 549 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:38:50,800::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:38:50,801::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:38:51,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:51,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913374 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:53,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:53,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914759 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:38:54,441::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:38:55,338::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:38:55,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:55,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.514496 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:38:57,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:38:58,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912052 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:00,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:00,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912914 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:39:00,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:39:00,679::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:39:00,700::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00269347 s, 138 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:39:00,702::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630223 s, 830 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:39:00,812::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:39:00,814::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:39:02,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:02,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918052 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:04,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:04,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559125 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:06,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:06,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912356 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:08,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:09,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091128 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:39:09,444::task::595::Storage.TaskManager.Task::(_updateState) Task=`17a5915e-99e1-4fc9-8093-7b1cabdd43e6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:39:09,444::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:39:09,448::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:39:09,448::task::1191::Storage.TaskManager.Task::(prepare) Task=`17a5915e-99e1-4fc9-8093-7b1cabdd43e6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:39:09,448::task::595::Storage.TaskManager.Task::(_updateState) Task=`17a5915e-99e1-4fc9-8093-7b1cabdd43e6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:39:09,448::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:39:09,448::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:39:09,448::task::993::Storage.TaskManager.Task::(_decref) Task=`17a5915e-99e1-4fc9-8093-7b1cabdd43e6`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:39:09,474::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:39:10,705::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:39:10,716::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:39:10,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00515185 s, 72.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:39:10,731::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00079264 s, 660 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:39:10,820::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:39:10,821::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:39:11,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:11,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913162 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:13,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:13,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913855 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:15,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:15,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913975 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:17,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:17,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912496 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:19,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:19,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913188 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:39:20,735::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:39:20,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:39:20,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00486713 s, 76.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:39:20,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626991 s, 834 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:39:20,833::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:39:20,834::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:39:21,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:21,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913267 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:23,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:23,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091322 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:39:24,490::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:39:25,371::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:39:25,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:26,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912909 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:28,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:28,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091523 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:30,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:30,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913499 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:39:30,761::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:39:30,773::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:39:30,789::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00061681 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:39:30,790::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000608584 s, 859 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:39:30,845::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:39:30,846::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:39:32,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:32,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911974 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:34,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:34,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915852 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:36,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:36,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914294 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:38,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:39,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503769 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:39:39,502::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:39:40,794::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:39:40,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:39:40,820::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681331 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:39:40,822::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688814 s, 759 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:39:40,857::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:39:40,858::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:39:41,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:41,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915993 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:43,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:43,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541272 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:45,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:45,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101591 s, 10.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:48,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:48,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916443 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:50,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:50,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.532355 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:39:50,827::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:39:50,838::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:39:50,849::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000732005 s, 507 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:39:50,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0010093 s, 518 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:39:50,871::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:39:50,872::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:39:52,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:52,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914728 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:39:54,514::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:39:54,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:55,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913455 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:39:55,405::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:39:57,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:57,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913845 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:39:59,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:39:59,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912895 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:40:00,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:40:00,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:40:00,877::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000715918 s, 518 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:40:00,879::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000798698 s, 655 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:40:00,890::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:40:00,891::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:40:01,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:01,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914136 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:03,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:03,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912574 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:05,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:05,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917387 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:07,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:07,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911647 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:40:09,520::task::595::Storage.TaskManager.Task::(_updateState) Task=`25a7a8eb-1308-48d1-93ba-78d3a5c6aeed`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:40:09,520::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:40:09,523::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:40:09,523::task::1191::Storage.TaskManager.Task::(prepare) Task=`25a7a8eb-1308-48d1-93ba-78d3a5c6aeed`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:40:09,523::task::595::Storage.TaskManager.Task::(_updateState) Task=`25a7a8eb-1308-48d1-93ba-78d3a5c6aeed`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:40:09,523::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:40:09,524::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:40:09,524::task::993::Storage.TaskManager.Task::(_decref) Task=`25a7a8eb-1308-48d1-93ba-78d3a5c6aeed`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:40:09,550::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:40:09,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:09,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913234 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:40:10,882::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:40:10,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:40:10,906::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632452 s, 587 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:40:10,911::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000749511 s, 698 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:40:10,918::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:40:10,920::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:40:11,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:11,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921638 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:13,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:14,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912951 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:16,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:16,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100303 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:18,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:18,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559373 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:40:20,914::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:40:20,928::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000608316 s, 610 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:40:20,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:40:20,948::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:40:20,950::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 16:40:20,951::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0037961 s, 138 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:20,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:21,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912028 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:23,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:23,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911981 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:40:24,562::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:40:25,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:25,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913116 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:40:25,438::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:40:27,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:27,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912652 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:29,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:29,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.516073 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:40:30,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:40:30,949::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000707391 s, 524 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:40:30,953::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:40:30,954::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 16:40:30,967::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:40:30,979::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630834 s, 829 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:31,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:32,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914638 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:34,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:34,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518738 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:36,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:36,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912475 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:38,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:39,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.520042 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:40:39,574::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 16:40:40,955::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:40:40,971::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623688 s, 595 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:40:40,974::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:40:40,978::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 16:40:40,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:40:40,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000612304 s, 854 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:41,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:41,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912506 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:43,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:43,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912435 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:45,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:45,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0988071 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:47,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:47,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912357 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:49,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:49,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918341 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:40:50,980::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:40:50,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00070278 s, 528 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:40:50,997::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 16:40:51,002::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:40:51,014::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 16:40:51,016::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006834 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:51,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:52,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542614 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:54,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:40:54,588::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:40:54,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911878 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:40:55,472::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:40:56,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:56,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911569 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:40:58,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:40:58,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911896 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:00,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:41:01,005::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:01,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967821 s, 10.6 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:41:01,022::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664122 s, 559 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:41:01,025::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 16:41:01,032::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:41:01,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000785979 s, 665 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:41:01,051::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:41:03,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:03,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914198 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:05,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:05,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913282 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:07,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:07,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.514267 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:41:09,592::task::595::Storage.TaskManager.Task::(_updateState) Task=`e432a933-5b19-4bab-abb0-356b8ac3f8b0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:41:09,592::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:41:09,595::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:41:09,595::task::1191::Storage.TaskManager.Task::(prepare) Task=`e432a933-5b19-4bab-abb0-356b8ac3f8b0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:41:09,595::task::595::Storage.TaskManager.Task::(_updateState) Task=`e432a933-5b19-4bab-abb0-356b8ac3f8b0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:41:09,595::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:41:09,596::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:41:09,596::task::993::Storage.TaskManager.Task::(_decref) Task=`e432a933-5b19-4bab-abb0-356b8ac3f8b0`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:41:09,622::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:41:09,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:09,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925509 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:41:11,016::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:41:11,017::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:41:11,035::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:41:11,047::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620575 s, 598 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:41:11,063::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:41:11,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006641 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:11,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:12,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914078 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:14,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:14,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914063 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:16,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:16,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100076 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:18,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:18,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918261 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:20,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:20,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912468 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:41:21,029::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:41:21,030::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:41:21,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:41:21,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000619576 s, 599 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:41:21,078::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:41:21,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637484 s, 820 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:22,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:22,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913964 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:41:24,635::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:41:24,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:24,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912457 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:41:25,507::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:41:26,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:27,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911501 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:29,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:29,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912775 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:41:31,039::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:41:31,040::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:41:31,065::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:41:31,076::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656262 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:41:31,093::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:41:31,105::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659974 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:31,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:31,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914364 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:33,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:33,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913628 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:35,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:35,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09153 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:37,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:38,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541015 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:41:39,650::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:41:40,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:40,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914319 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:41:41,052::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:41:41,053::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:41:41,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:41:41,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652759 s, 568 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:41:41,109::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:41:41,121::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069463 s, 753 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:42,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:42,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504318 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:44,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:44,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913598 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:46,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:46,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913635 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:48,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:49,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912915 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:41:51,064::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:41:51,065::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:41:51,094::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:51,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:41:51,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000696852 s, 532 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:41:51,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:51,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103145 s, 9.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:41:51,228::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00515113 s, 102 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:53,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:53,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911987 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:41:54,662::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:41:55,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:41:55,540::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:41:55,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542535 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:41:57,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:41:58,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912553 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:00,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:00,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091261 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:42:01,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:42:01,077::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:42:01,124::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:42:01,135::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000720524 s, 515 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:42:01,238::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:42:01,250::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676231 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:02,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:02,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912889 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:04,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:04,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911954 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:06,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:06,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917961 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:08,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:08,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918396 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:42:09,665::task::595::Storage.TaskManager.Task::(_updateState) Task=`49164228-6814-43d0-bff1-e563653b9b4e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:42:09,665::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:42:09,669::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:42:09,669::task::1191::Storage.TaskManager.Task::(prepare) Task=`49164228-6814-43d0-bff1-e563653b9b4e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:42:09,669::task::595::Storage.TaskManager.Task::(_updateState) Task=`49164228-6814-43d0-bff1-e563653b9b4e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:42:09,670::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:42:09,670::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:42:09,670::task::993::Storage.TaskManager.Task::(_decref) Task=`49164228-6814-43d0-bff1-e563653b9b4e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:42:09,697::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:42:10,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:10,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912866 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:42:11,086::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:42:11,087::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:42:11,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:42:11,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687818 s, 539 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:42:11,253::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:42:11,265::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675923 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:12,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:12,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912394 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:14,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:14,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913828 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:16,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:17,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912468 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:19,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:19,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912195 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:42:21,098::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:42:21,099::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:42:21,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:42:21,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629129 s, 590 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:21,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:42:21,332::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:42:21,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000738649 s, 708 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:21,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108644 s, 9.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:23,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:23,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912987 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:42:24,709::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:42:25,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:42:25,573::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:42:25,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913498 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:27,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:27,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913855 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:29,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:29,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091159 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:42:31,111::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:42:31,112::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:42:31,169::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:42:31,180::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00064905 s, 572 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:42:31,354::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:42:31,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621846 s, 841 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:31,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:32,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543308 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:34,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:34,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.509482 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:36,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:37,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540596 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:39,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:39,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912235 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:42:39,722::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:42:41,117::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:42:41,118::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:42:41,184::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:42:41,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686015 s, 541 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:42:41,370::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:42:41,381::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665706 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:41,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:41,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914942 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:43,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:43,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913401 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:45,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:46,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541449 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:48,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:48,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912443 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:50,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:50,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913038 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:42:51,131::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:42:51,132::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:42:51,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:42:51,209::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650682 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:42:51,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:42:51,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682046 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:52,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:52,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912516 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:42:54,736::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:42:54,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:54,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911366 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:42:55,608::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:42:57,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:57,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091483 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:42:59,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:42:59,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913433 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:43:01,142::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:43:01,143::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:43:01,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:43:01,224::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641194 s, 579 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:01,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:43:01,399::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:43:01,803::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:43:01,805::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Dummy-49::DEBUG::2014-07-14 16:43:01,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55495 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:43:01,811::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:43:01,817::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:43:01,819::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:43:01,820::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:43:01,823::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:43:01,836::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629143 s, 831 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:03,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:03,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918696 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:05,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:06,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913913 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:08,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:08,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910917 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:43:09,739::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d89d8f4-3bc2-492a-8d8c-cd832ff3887d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:43:09,739::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:43:09,742::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:43:09,742::task::1191::Storage.TaskManager.Task::(prepare) Task=`1d89d8f4-3bc2-492a-8d8c-cd832ff3887d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:43:09,742::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d89d8f4-3bc2-492a-8d8c-cd832ff3887d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:43:09,743::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:43:09,743::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:43:09,743::task::993::Storage.TaskManager.Task::(_decref) Task=`1d89d8f4-3bc2-492a-8d8c-cd832ff3887d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:43:09,770::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:43:10,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:10,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540322 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:43:11,152::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:43:11,154::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:43:11,228::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:43:11,240::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618104 s, 600 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:43:11,845::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:43:11,857::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648132 s, 807 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:12,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:12,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913028 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:14,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:14,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913625 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:17,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:17,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914675 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:19,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:19,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911878 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:43:21,162::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:43:21,163::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:43:21,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:43:21,250::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:43:21,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000762621 s, 486 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:21,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.095318 s, 10.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:43:21,860::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:43:21,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000752239 s, 695 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:23,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:23,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911715 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:43:24,783::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:43:25,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:25,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913245 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:43:25,641::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:43:27,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:27,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913073 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:29,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:29,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913219 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:43:31,172::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:43:31,173::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:43:31,346::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:43:31,359::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:43:31,360::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:43:31,363::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:43:31,365::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:43:31,365::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:43:31,365::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:43:31,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:43:31,378::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664697 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:31,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:43:31,932::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:31,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104217 s, 9.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:43:31,951::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00582703 s, 89.8 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:33,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:34,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913515 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:36,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:36,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913025 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:38,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:38,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.090944 s, 11.3 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:43:39,795::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:43:40,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:40,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913522 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:43:41,185::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:43:41,186::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:43:41,382::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:43:41,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650126 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:43:41,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:43:41,969::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667056 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:42,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:42,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912841 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:44,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:44,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913612 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:46,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:46,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:48,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:48,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911989 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:50,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:51,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915035 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:43:51,197::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:43:51,199::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:43:51,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:43:51,408::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000717168 s, 517 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:43:51,973::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:43:51,985::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670544 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:53,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:53,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914309 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:43:54,808::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:43:55,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:55,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912767 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:43:55,672::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:43:57,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:57,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09113 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:43:59,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:43:59,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547859 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:44:01,209::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:44:01,211::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:44:01,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:44:01,422::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674349 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:01,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:44:02,501::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:44:02,514::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681831 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:02,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.564838 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:04,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:05,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547058 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:07,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:07,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911503 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:09,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:09,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910044 s, 11.3 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:44:09,812::task::595::Storage.TaskManager.Task::(_updateState) Task=`7569e450-49b2-49c9-a901-4f3fdbca81cc`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:44:09,812::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:44:09,815::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:44:09,815::task::1191::Storage.TaskManager.Task::(prepare) Task=`7569e450-49b2-49c9-a901-4f3fdbca81cc`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:44:09,815::task::595::Storage.TaskManager.Task::(_updateState) Task=`7569e450-49b2-49c9-a901-4f3fdbca81cc`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:44:09,815::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:44:09,815::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:44:09,816::task::993::Storage.TaskManager.Task::(_decref) Task=`7569e450-49b2-49c9-a901-4f3fdbca81cc`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:44:09,848::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:44:11,221::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:44:11,222::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:44:11,425::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:44:11,437::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000611927 s, 606 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:11,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:11,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912284 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:44:12,522::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:44:12,535::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000613536 s, 852 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:13,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:13,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913677 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:15,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:16,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101568 s, 10.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:18,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:18,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914394 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:20,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:20,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912768 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:44:21,234::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:44:21,235::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:44:21,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:44:21,452::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620567 s, 598 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:22,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:44:22,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:44:22,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068864 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:22,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.53374 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:44:24,861::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:44:25,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:25,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.53844 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:44:25,706::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:44:27,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:28,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539709 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:30,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:30,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913243 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:44:31,243::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:44:31,244::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:44:31,456::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:44:31,467::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679642 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:32,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:32,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912041 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:44:32,993::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:44:33,005::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070312 s, 744 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:34,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:34,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912903 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:36,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:36,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912509 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:38,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:38,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915295 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:44:39,872::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:44:40,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:44:41,252::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:44:41,254::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:44:41,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548934 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:44:41,471::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:44:41,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683985 s, 542 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:44:43,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:44:43,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000711158 s, 735 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:43,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:43,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545442 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:45,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:46,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0972233 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:48,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:48,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913267 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:50,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:50,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911566 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:44:51,265::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:44:51,266::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:44:51,486::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:44:51,497::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647137 s, 573 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:52,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:52,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0928509 s, 11.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:44:53,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:44:53,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000738199 s, 708 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:54,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:54,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911603 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:44:54,884::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:44:55,739::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:44:56,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:56,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920778 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:44:58,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:44:59,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548225 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:45:01,277::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:45:01,278::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:45:01,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:01,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915053 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:45:01,501::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:45:01,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671852 s, 552 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:45:03,040::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:45:03,052::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000715221 s, 731 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:03,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:03,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919043 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:05,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:05,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913643 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:07,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:07,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913126 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:09,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:45:09,887::task::595::Storage.TaskManager.Task::(_updateState) Task=`d86fcfac-4cce-4b95-be44-55de9a88c28f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:45:09,887::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:45:10,339::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:45:10,340::task::1191::Storage.TaskManager.Task::(prepare) Task=`d86fcfac-4cce-4b95-be44-55de9a88c28f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:45:10,341::task::595::Storage.TaskManager.Task::(_updateState) Task=`d86fcfac-4cce-4b95-be44-55de9a88c28f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:45:10,341::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:45:10,342::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:45:10,343::task::993::Storage.TaskManager.Task::(_decref) Task=`d86fcfac-4cce-4b95-be44-55de9a88c28f`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 16:45:10,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555064 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:45:10,374::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:45:11,289::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:45:11,291::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:45:11,516::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:45:11,528::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617217 s, 601 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:12,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:12,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541959 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:45:13,056::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:45:13,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690974 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:14,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:15,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50799 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:17,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:17,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913563 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:19,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:19,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913426 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:45:21,302::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:45:21,303::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:45:21,531::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:45:21,543::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643485 s, 577 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:21,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:21,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912803 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:45:23,072::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:45:23,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670646 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:23,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:23,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912387 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:45:25,387::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:45:25,779::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:45:25,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:26,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0987692 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:28,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:28,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912471 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:30,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:30,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0995375 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:45:31,314::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:45:31,315::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:45:31,547::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:45:31,558::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705098 s, 526 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:32,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:32,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912301 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:45:33,088::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:45:33,100::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000722963 s, 723 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:34,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:34,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091346 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:36,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:36,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091825 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:38,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:39,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555873 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:45:40,400::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:45:41,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:45:41,321::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:45:41,322::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:45:41,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914193 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:45:41,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:45:41,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632189 s, 587 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:45:43,104::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:45:43,115::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685133 s, 763 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:43,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:43,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542845 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:45,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:46,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101333 s, 10.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:48,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:48,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911988 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:50,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:50,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911673 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:45:51,333::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:45:51,334::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:45:51,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:45:51,588::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669765 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:52,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:52,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.180789 s, 5.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:45:53,119::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:45:53,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677558 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:54,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:55,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54154 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:45:55,413::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:45:55,811::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:45:57,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:57,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912527 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:45:59,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:45:59,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09127 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:46:01,345::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:46:01,346::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:46:01,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:01,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091308 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:46:01,591::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:46:01,603::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650694 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:46:03,135::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:46:03,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675949 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:03,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:03,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913743 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:05,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:06,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507328 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:08,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:08,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912914 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:10,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:46:10,417::task::595::Storage.TaskManager.Task::(_updateState) Task=`478f0dd9-f37b-4cef-a5b5-c77deab765c4`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:46:10,417::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:46:10,453::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:46:10,455::task::1191::Storage.TaskManager.Task::(prepare) Task=`478f0dd9-f37b-4cef-a5b5-c77deab765c4`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:46:10,457::task::595::Storage.TaskManager.Task::(_updateState) Task=`478f0dd9-f37b-4cef-a5b5-c77deab765c4`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:46:10,458::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:46:10,459::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:46:10,461::task::993::Storage.TaskManager.Task::(_decref) Task=`478f0dd9-f37b-4cef-a5b5-c77deab765c4`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 16:46:10,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.105764 s, 9.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:46:10,494::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:46:11,356::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:46:11,357::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:46:11,607::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:46:11,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620155 s, 598 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:12,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:12,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912611 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:46:13,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:46:13,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691681 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:14,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:14,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913918 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:16,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:16,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911202 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:18,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:19,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912495 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:21,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:21,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913626 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:46:21,369::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:46:21,370::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:46:21,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:46:21,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625517 s, 593 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:23,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:46:23,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:46:23,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064704 s, 808 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:23,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106185 s, 9.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:25,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:25,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917013 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:46:25,508::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:46:25,849::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:46:27,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:27,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543164 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:30,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:30,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.505381 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:46:31,377::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:46:31,379::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:46:31,637::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:46:31,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669205 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:32,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:32,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911819 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:46:33,305::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:46:33,317::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682006 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:34,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:34,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920799 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:36,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:36,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913551 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:38,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:38,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912347 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:46:40,519::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:46:41,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:41,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915784 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:46:41,389::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:46:41,391::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:46:41,652::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:46:41,663::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699657 s, 530 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:43,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:43,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912224 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:46:43,322::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:46:43,333::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686893 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:45,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:45,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913524 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:47,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:47,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507365 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:49,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:50,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09131 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:46:51,402::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:46:51,403::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:46:51,668::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:46:51,679::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692799 s, 536 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:52,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:52,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913391 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:46:53,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:46:53,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664234 s, 787 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:54,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:54,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912411 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:46:55,531::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:46:55,883::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:46:56,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:56,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912316 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:46:58,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:46:58,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912004 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:00,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:00,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909741 s, 11.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:47:01,413::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:47:01,414::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:47:01,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:47:01,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691151 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:02,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:02,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913631 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:47:03,353::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:47:03,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673893 s, 776 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:04,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:04,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913102 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:06,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:06,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913688 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:08,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:09,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914671 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:47:10,534::task::595::Storage.TaskManager.Task::(_updateState) Task=`140826c7-b6d6-4817-9d42-8c8907274a0f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:47:10,534::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:47:10,537::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:47:10,537::task::1191::Storage.TaskManager.Task::(prepare) Task=`140826c7-b6d6-4817-9d42-8c8907274a0f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:47:10,537::task::595::Storage.TaskManager.Task::(_updateState) Task=`140826c7-b6d6-4817-9d42-8c8907274a0f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:47:10,537::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:47:10,538::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:47:10,538::task::993::Storage.TaskManager.Task::(_decref) Task=`140826c7-b6d6-4817-9d42-8c8907274a0f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:47:10,564::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:47:11,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:11,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913672 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:47:11,425::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:47:11,427::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:47:11,697::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:47:11,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652384 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:13,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:13,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911582 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:47:13,368::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:47:13,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006615 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:15,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:15,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.133813 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:17,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:17,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916347 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:19,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:20,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.533098 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:47:21,435::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:47:21,436::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:47:21,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:47:21,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676492 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:22,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:22,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913455 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:47:23,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:47:23,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000704889 s, 742 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:24,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:24,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.093327 s, 11.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:47:25,575::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:47:25,916::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:47:26,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:26,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911758 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:28,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:28,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912097 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:30,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:30,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912326 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:47:31,447::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:47:31,448::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:47:31,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:47:31,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000712728 s, 521 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:32,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:32,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912471 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:47:33,400::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:47:33,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637575 s, 820 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:34,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:34,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911493 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:36,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:37,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913468 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:39,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:39,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913377 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:47:40,587::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:47:41,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:41,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913967 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:47:41,460::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:47:41,461::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:47:41,744::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:47:41,755::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680977 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:43,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:47:43,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:47:43,844::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000748412 s, 699 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:43,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.526099 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:45,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:46,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549978 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:48,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:49,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504375 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:51,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:51,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911977 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:47:51,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:47:51,476::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:47:51,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:47:51,770::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660134 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:53,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:53,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912833 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:47:53,852::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:47:53,863::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668499 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:55,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:55,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912959 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:47:55,600::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >VM Channels Listener::DEBUG::2014-07-14 16:47:55,948::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:47:57,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:47:57,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518178 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:47:59,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:00,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911161 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:48:01,487::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:48:01,488::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:48:01,773::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:48:01,785::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645124 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:02,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:02,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912218 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:48:03,866::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:48:03,879::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:48:03,881::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:48:03,884::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:48:03,885::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:48:03,885::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:48:03,885::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:48:03,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:48:03,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640128 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:04,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:04,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912248 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:06,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:06,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.096559 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:08,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:08,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091257 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:10,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:48:10,603::task::595::Storage.TaskManager.Task::(_updateState) Task=`babcf338-443d-4279-81b5-377b05b5fa14`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:48:10,603::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:48:11,065::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:48:11,066::task::1191::Storage.TaskManager.Task::(prepare) Task=`babcf338-443d-4279-81b5-377b05b5fa14`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:48:11,066::task::595::Storage.TaskManager.Task::(_updateState) Task=`babcf338-443d-4279-81b5-377b05b5fa14`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:48:11,067::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:48:11,068::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:48:11,069::task::993::Storage.TaskManager.Task::(_decref) Task=`babcf338-443d-4279-81b5-377b05b5fa14`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 16:48:11,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546121 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:48:11,104::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:48:11,499::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:48:11,500::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:48:11,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:48:11,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659984 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:13,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:13,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542747 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:48:13,902::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:48:13,914::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640311 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:15,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:15,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967684 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:17,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:17,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912934 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:20,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:20,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913742 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:48:21,510::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:48:21,511::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:48:21,803::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:48:21,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663614 s, 559 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:22,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:22,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913981 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:48:23,917::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:48:23,928::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688844 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:24,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:24,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913353 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:48:25,982::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:48:26,117::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:48:26,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:26,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914493 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:28,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:28,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913962 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:30,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:30,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913154 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:48:31,522::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:48:31,523::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:48:31,817::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:48:31,832::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:48:31,833::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:48:31,836::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:48:31,837::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:48:31,838::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:48:31,838::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:48:31,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:48:31,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654531 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:32,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:32,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912276 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:48:33,933::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:48:33,945::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636221 s, 822 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:34,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:34,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912955 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:36,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:37,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920839 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:39,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:39,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919586 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:48:41,130::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:48:41,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:41,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914229 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:48:41,534::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:48:41,535::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:48:41,855::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:48:41,867::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661073 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:43,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:43,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915911 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:48:43,949::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:48:43,960::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000714312 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:45,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:45,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915576 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:47,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:48,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557726 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:50,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:50,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913093 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:48:51,547::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:48:51,548::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:48:51,870::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:48:51,881::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647109 s, 573 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:52,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:52,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911988 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:48:53,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:48:53,975::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665741 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:54,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:54,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914755 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:48:56,016::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:48:56,143::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:48:56,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:56,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912435 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:48:58,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:48:58,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091392 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:00,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:00,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912523 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:49:01,555::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:49:01,556::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:49:01,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:49:01,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663367 s, 559 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:02,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:02,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913663 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:49:03,979::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:49:03,991::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000654613 s, 799 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:04,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:05,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921214 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:07,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:07,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913525 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:09,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:09,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912701 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:49:11,146::task::595::Storage.TaskManager.Task::(_updateState) Task=`869c9479-1be2-43f6-bdfd-e4b35cc7237f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:49:11,146::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:49:11,152::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:49:11,152::task::1191::Storage.TaskManager.Task::(prepare) Task=`869c9479-1be2-43f6-bdfd-e4b35cc7237f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:49:11,152::task::595::Storage.TaskManager.Task::(_updateState) Task=`869c9479-1be2-43f6-bdfd-e4b35cc7237f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:49:11,152::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:49:11,152::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:49:11,152::task::993::Storage.TaskManager.Task::(_decref) Task=`869c9479-1be2-43f6-bdfd-e4b35cc7237f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:49:11,179::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:49:11,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:11,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911632 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:49:11,563::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:49:11,565::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:49:11,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:49:11,923::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000638777 s, 581 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:13,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:13,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502595 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:49:13,995::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:49:14,008::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658943 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:15,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:16,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938124 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:18,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:18,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913263 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:20,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:20,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911841 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:49:21,575::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:49:21,577::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:49:21,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:49:21,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000602095 s, 616 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:22,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:22,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913654 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:49:24,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:49:24,023::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006774 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:24,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:24,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911935 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:49:26,049::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:49:26,190::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:49:26,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:26,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916827 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:28,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:28,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912251 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:30,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:31,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913616 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:49:31,588::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:49:31,589::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:49:31,940::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:49:31,952::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674519 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:33,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:33,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511775 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:49:34,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:49:34,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656332 s, 797 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:35,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:35,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913919 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:37,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:38,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545491 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:40,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:40,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556389 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:49:41,203::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:49:41,599::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:49:41,600::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:49:41,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:49:41,967::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00095341 s, 389 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:42,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:42,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912302 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:49:44,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:49:44,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667328 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:44,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:45,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912683 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:47,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:47,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917031 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:49,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:49,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540377 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:49:51,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:49:51,610::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:49:51,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:51,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914172 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:49:51,971::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:49:51,982::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679404 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:53,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:53,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913634 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:49:54,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:49:54,070::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693287 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:49:55,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:56,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913157 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:49:56,082::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:49:56,215::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:49:58,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:49:58,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511634 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:00,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:00,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914088 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:50:01,621::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:50:01,622::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:50:01,986::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:50:01,997::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674261 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:02,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:02,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912013 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:50:04,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:50:04,085::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668104 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:04,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:04,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912149 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:06,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:07,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913458 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:09,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:09,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911401 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:50:11,218::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6ac87bb-971a-45e3-a7f4-8ee9a424306e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:50:11,218::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Dummy-49::DEBUG::2014-07-14 16:50:11,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::INFO::2014-07-14 16:50:11,232::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:50:11,233::task::1191::Storage.TaskManager.Task::(prepare) Task=`b6ac87bb-971a-45e3-a7f4-8ee9a424306e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:50:11,233::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6ac87bb-971a-45e3-a7f4-8ee9a424306e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:50:11,233::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:50:11,233::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:50:11,233::task::993::Storage.TaskManager.Task::(_decref) Task=`b6ac87bb-971a-45e3-a7f4-8ee9a424306e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:50:11,262::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:50:11,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912654 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:50:11,633::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:50:11,636::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:50:12,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:50:12,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660576 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:13,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:13,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091256 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:50:14,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:50:14,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637902 s, 820 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:15,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:15,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913208 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:17,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:17,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918332 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:19,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:19,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913365 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:50:21,640::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:50:21,641::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:50:21,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:21,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922042 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:50:22,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:50:22,029::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655455 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:23,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:24,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922077 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:50:24,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:50:24,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695921 s, 752 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:50:26,121::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:50:26,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:26,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912006 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:50:26,274::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:50:28,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:28,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913074 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:30,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:30,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923597 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:50:31,652::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:50:31,653::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:50:32,032::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:50:32,044::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00072255 s, 513 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:32,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:32,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912241 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:50:34,135::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:50:34,147::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066417 s, 787 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:34,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:34,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912864 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:36,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:37,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541695 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:39,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:39,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541467 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:50:41,286::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:50:41,661::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:50:41,662::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:50:41,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:41,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912188 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:50:42,048::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:50:42,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675927 s, 549 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:43,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:44,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912652 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:50:44,151::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:50:44,162::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068934 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:46,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:46,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101244 s, 10.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:48,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:48,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911226 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:50,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:50,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.130746 s, 7.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:50:51,670::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:50:51,671::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:50:52,062::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:50:52,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658368 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:52,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:52,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912273 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:50:54,166::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:50:54,178::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699334 s, 748 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:54,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:54,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912266 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:50:56,155::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:50:56,297::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:50:56,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:56,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913195 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:50:59,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:50:59,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091298 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:01,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:01,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913398 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:51:01,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:51:01,683::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:51:02,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:51:02,088::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649807 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:03,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:03,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913913 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:51:04,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:51:04,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680883 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:05,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:05,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548211 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:07,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:08,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913002 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:10,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:10,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539913 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:51:11,300::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fd5f5a3-5960-44cd-b587-15844d2b6804`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:51:11,300::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:51:11,303::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:51:11,303::task::1191::Storage.TaskManager.Task::(prepare) Task=`1fd5f5a3-5960-44cd-b587-15844d2b6804`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:51:11,304::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fd5f5a3-5960-44cd-b587-15844d2b6804`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:51:11,304::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:51:11,304::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:51:11,304::task::993::Storage.TaskManager.Task::(_decref) Task=`1fd5f5a3-5960-44cd-b587-15844d2b6804`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:51:11,329::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:51:11,690::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:51:11,691::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:51:12,092::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:51:12,104::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618987 s, 599 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:12,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:12,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913587 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:51:14,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:51:14,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00096639 s, 541 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:14,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:14,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915701 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:16,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:16,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921876 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:18,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:19,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912687 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:21,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:21,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912448 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:51:21,703::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:51:21,704::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:51:22,151::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:51:22,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654676 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:23,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:23,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914597 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:51:24,220::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:51:24,231::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673421 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:25,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:25,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915693 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:51:26,189::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:51:26,341::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:51:27,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:27,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519356 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:30,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:30,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914646 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:51:31,715::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:51:31,716::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:51:32,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:51:32,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:51:32,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000634582 s, 585 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:32,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.111128 s, 9.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:51:34,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:51:34,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668703 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:34,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:34,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091156 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:36,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:36,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913363 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:38,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:38,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092158 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:40,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:40,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914966 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:51:41,352::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:51:41,722::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:51:41,723::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:51:42,242::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:51:42,254::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693864 s, 535 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:42,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:43,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548152 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:51:44,252::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:51:44,264::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665391 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:45,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:45,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914532 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:47,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:48,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39777 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:50,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:51,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543343 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:51:51,734::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:51:51,736::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:51:52,257::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:51:52,268::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066906 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:53,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:53,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919287 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:51:54,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:51:54,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699163 s, 748 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:55,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:55,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914495 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:51:56,221::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:51:56,363::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:51:57,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:51:57,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920092 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:51:59,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:00,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558987 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:52:01,742::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:52:01,743::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:52:02,271::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:52:02,283::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656067 s, 565 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:02,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:02,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912718 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:52:04,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:52:04,294::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690507 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:04,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:04,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914705 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:06,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:06,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915761 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:08,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:08,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914785 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:10,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:10,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913065 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:52:11,366::task::595::Storage.TaskManager.Task::(_updateState) Task=`f56e179b-17ae-457d-884d-983a88b8a201`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:52:11,366::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:52:11,370::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:52:11,370::task::1191::Storage.TaskManager.Task::(prepare) Task=`f56e179b-17ae-457d-884d-983a88b8a201`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:52:11,370::task::595::Storage.TaskManager.Task::(_updateState) Task=`f56e179b-17ae-457d-884d-983a88b8a201`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:52:11,370::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:52:11,370::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:52:11,371::task::993::Storage.TaskManager.Task::(_decref) Task=`f56e179b-17ae-457d-884d-983a88b8a201`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:52:11,398::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:52:11,752::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:52:11,753::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:52:12,286::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:52:12,298::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662224 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:12,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:13,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913316 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:52:14,298::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:52:14,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675912 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:15,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:15,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091425 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:17,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:17,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543424 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:19,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:19,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091349 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:52:21,764::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:52:21,765::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:52:21,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:21,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913513 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:52:22,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:52:22,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648281 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:23,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:24,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915283 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:52:24,314::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:52:24,325::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706619 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:26,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:26,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915135 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:52:26,255::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:52:26,410::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:52:28,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:28,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912912 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:30,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:30,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912526 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:52:31,776::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:52:31,778::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:52:32,316::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:52:32,328::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690116 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:32,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:32,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913203 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:52:34,329::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:52:34,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647301 s, 808 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:34,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:34,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912954 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:36,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:36,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913764 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:38,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:38,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.098269 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:40,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:41,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913017 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:52:41,422::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:52:41,789::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:52:41,790::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:52:42,336::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:52:42,348::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645144 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:43,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:43,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554286 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:52:44,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:52:44,356::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707967 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:45,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:47,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37984 s, 742 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:49,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:49,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550488 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:51,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:52:51,801::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:52:51,802::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:52:52,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54332 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:52:52,351::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:52:52,362::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670285 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:54,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:54,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912601 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:52:54,360::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:52:54,372::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660955 s, 791 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:52:56,288::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:52:56,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:52:56,434::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:52:56,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543718 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:52:58,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:52:59,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541956 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:01,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:53:01,813::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:53:01,814::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:53:01,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542087 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:53:02,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:53:02,377::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000736152 s, 504 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:04,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:04,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916496 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:53:04,374::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:53:04,837::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:53:04,838::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:53:04,841::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:53:04,843::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:53:04,843::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:53:04,843::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:53:04,844::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:53:04,856::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000654515 s, 799 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:06,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:06,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914361 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:08,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:08,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912955 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:10,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:10,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.52095 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:53:11,438::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc6f4e90-503c-43fd-b7e2-b951fddf6827`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:53:11,438::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:53:11,441::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:53:11,441::task::1191::Storage.TaskManager.Task::(prepare) Task=`fc6f4e90-503c-43fd-b7e2-b951fddf6827`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:53:11,442::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc6f4e90-503c-43fd-b7e2-b951fddf6827`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:53:11,442::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:53:11,442::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:53:11,442::task::993::Storage.TaskManager.Task::(_decref) Task=`fc6f4e90-503c-43fd-b7e2-b951fddf6827`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:53:11,469::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:53:11,822::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:53:11,823::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:53:12,383::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:53:12,397::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000602449 s, 616 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:12,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:13,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909804 s, 11.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:53:14,860::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:53:14,872::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621854 s, 841 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:15,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:15,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915144 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:17,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:17,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.514702 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:19,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:19,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912381 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:21,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:53:21,832::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:53:21,833::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:53:21,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913511 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:53:22,404::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:53:22,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674042 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:23,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:24,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912583 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:53:24,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:53:24,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685741 s, 763 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:26,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:26,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912765 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:53:26,320::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:53:26,481::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:53:28,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:28,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913103 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:30,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:30,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091292 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:53:31,843::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:53:31,844::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:53:32,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:53:32,422::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Dummy-49::DEBUG::2014-07-14 16:53:32,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.53508 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:53:32,943::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:53:32,945::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:53:32,950::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:53:32,954::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:53:32,955::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:53:32,956::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:53:32,959::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:53:32,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686433 s, 540 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:53:34,892::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:53:34,905::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006385 s, 819 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:34,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:35,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913473 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:37,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:37,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544519 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:39,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:39,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915218 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:53:41,499::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:53:41,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:53:41,851::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:53:41,852::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:53:41,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913769 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:53:42,979::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:53:42,990::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:43,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:44,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091264 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:53:44,909::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:53:44,921::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694188 s, 753 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:46,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:46,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968451 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:48,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:48,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912038 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:50,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:50,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.523957 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:53:51,860::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:53:51,861::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:53:52,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:53:53,041::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:53:53,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000694468 s, 534 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:53,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.107722 s, 9.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:53:54,925::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:53:54,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662377 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:55,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:55,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914372 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:53:56,352::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:53:56,511::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:53:57,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:57,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914343 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:53:59,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:53:59,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914111 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:01,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:54:01,873::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:54:01,874::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:54:01,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542134 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:54:03,062::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:54:03,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689027 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:03,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:04,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540008 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:54:04,940::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:54:04,952::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067103 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:06,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:06,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911286 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:08,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:08,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913326 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:10,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:10,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914327 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:54:11,514::task::595::Storage.TaskManager.Task::(_updateState) Task=`5408feee-e500-4cbd-a2ea-06fbed2e08f1`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:54:11,514::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:54:11,517::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:54:11,518::task::1191::Storage.TaskManager.Task::(prepare) Task=`5408feee-e500-4cbd-a2ea-06fbed2e08f1`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:54:11,518::task::595::Storage.TaskManager.Task::(_updateState) Task=`5408feee-e500-4cbd-a2ea-06fbed2e08f1`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:54:11,518::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:54:11,518::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:54:11,518::task::993::Storage.TaskManager.Task::(_decref) Task=`5408feee-e500-4cbd-a2ea-06fbed2e08f1`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:54:11,544::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:54:11,882::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:54:11,883::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:54:12,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:13,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913225 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:54:13,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:54:13,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635695 s, 584 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:54:14,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:54:14,968::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651496 s, 803 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:15,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:15,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924383 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:17,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:17,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913646 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:19,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:19,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919713 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:21,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:21,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913509 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:54:21,893::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:54:21,894::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:54:23,095::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:54:23,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659905 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:23,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:23,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091412 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:54:24,971::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:54:24,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683078 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:25,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:25,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091312 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:54:26,385::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:54:26,557::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:54:27,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:27,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914369 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:29,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:30,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911894 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:54:31,903::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:54:31,904::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:54:32,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:32,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915127 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:54:33,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:54:33,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00070639 s, 525 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:34,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:34,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913516 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:54:34,988::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:54:35,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639935 s, 817 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:36,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:36,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913928 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:38,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:38,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912721 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:40,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:40,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915016 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:54:41,569::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:54:41,914::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:54:41,915::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:54:42,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:42,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091505 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:54:43,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:54:43,137::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658464 s, 563 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:44,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:44,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914157 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:54:45,019::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:54:45,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669249 s, 781 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:46,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:47,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55773 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:49,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:49,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921752 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:51,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:51,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09135 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:54:51,927::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:54:51,928::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:54:53,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:54:53,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067377 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:53,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:54,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540417 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:54:55,035::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:54:55,046::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656642 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:56,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:54:56,419::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:54:56,581::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:54:57,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36644 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:54:59,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:54:59,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913185 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:01,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:01,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923565 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:55:01,939::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:55:01,940::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:55:03,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:55:03,168::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000698363 s, 531 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:03,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:03,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921223 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:55:05,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:55:05,062::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666708 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:05,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:06,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540311 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:08,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:08,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913558 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:10,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:11,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547495 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:55:11,584::task::595::Storage.TaskManager.Task::(_updateState) Task=`1937a861-8642-4ca3-b42d-5636577a479c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:55:11,584::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:55:11,587::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:55:11,588::task::1191::Storage.TaskManager.Task::(prepare) Task=`1937a861-8642-4ca3-b42d-5636577a479c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:55:11,588::task::595::Storage.TaskManager.Task::(_updateState) Task=`1937a861-8642-4ca3-b42d-5636577a479c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:55:11,588::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:55:11,588::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:55:11,588::task::993::Storage.TaskManager.Task::(_decref) Task=`1937a861-8642-4ca3-b42d-5636577a479c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:55:11,616::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:55:11,951::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:55:11,952::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:55:13,172::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:55:13,184::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623228 s, 595 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:13,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:13,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913289 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:55:15,066::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:55:15,078::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688624 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:15,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:15,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912663 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:17,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:17,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913167 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:19,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:20,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549313 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:55:21,963::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:55:21,964::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:55:22,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:22,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091383 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:55:23,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:55:23,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659947 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:24,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:24,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915036 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:55:25,082::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:55:25,093::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672251 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:26,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:55:26,451::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:55:26,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914112 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:55:26,628::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:55:28,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:28,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911184 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:30,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:30,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917966 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:55:31,975::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:55:31,976::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:55:32,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:32,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914689 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:55:33,202::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:55:33,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652416 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:34,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:34,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913813 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:55:35,098::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:55:35,109::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629932 s, 830 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:36,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:37,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091274 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:39,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:39,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913248 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:41,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:41,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913842 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:55:41,640::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:55:41,987::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:55:41,989::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:55:43,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:55:43,228::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643033 s, 577 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:43,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:43,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910502 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:55:45,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:55:45,125::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665406 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:45,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:45,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914558 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:47,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:47,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913637 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:49,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:49,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913595 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:51,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:51,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913869 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:55:52,000::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:55:52,001::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:55:53,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:55:53,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000808508 s, 459 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:53,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:54,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912965 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:55:55,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:55:55,142::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686931 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:55:56,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:56,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914066 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:55:56,485::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:55:56,661::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:55:58,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:55:58,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916395 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:00,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:00,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913283 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:56:02,012::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:56:02,013::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:56:02,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:03,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549404 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:56:03,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:56:03,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677223 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:05,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:05,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914026 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:56:05,147::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:56:05,159::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695205 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:07,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:07,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913441 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:09,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:09,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912185 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:11,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:11,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914077 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:56:11,664::task::595::Storage.TaskManager.Task::(_updateState) Task=`06c09bd7-6bed-4bc4-b04c-cd8a05d75d1c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:56:11,664::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:56:11,667::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:56:11,668::task::1191::Storage.TaskManager.Task::(prepare) Task=`06c09bd7-6bed-4bc4-b04c-cd8a05d75d1c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:56:11,668::task::595::Storage.TaskManager.Task::(_updateState) Task=`06c09bd7-6bed-4bc4-b04c-cd8a05d75d1c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:56:11,668::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:56:11,668::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:56:11,668::task::993::Storage.TaskManager.Task::(_decref) Task=`06c09bd7-6bed-4bc4-b04c-cd8a05d75d1c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:56:11,693::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:56:12,017::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:56:12,018::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:56:13,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:56:13,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653159 s, 568 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:13,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:13,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914052 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:56:15,167::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:56:15,179::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000617772 s, 847 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:15,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:15,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968165 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:17,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:18,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913744 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:20,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:20,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912436 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:56:22,029::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:56:22,030::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:56:22,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:22,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917431 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:56:23,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:56:23,294::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656753 s, 565 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:24,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:24,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913732 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:56:25,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:56:25,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673533 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:26,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:26,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913696 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:56:26,518::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:56:26,708::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:56:28,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:28,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913221 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:30,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:31,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50725 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:56:32,042::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:56:32,043::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:56:33,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:33,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914166 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:56:33,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:56:33,309::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656404 s, 565 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:56:35,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:56:35,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068291 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:35,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:35,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543931 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:37,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:37,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917731 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:39,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:40,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914608 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:56:41,721::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:56:42,049::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:56:42,051::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:56:42,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:42,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915081 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:56:43,312::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:56:43,324::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679954 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:44,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:44,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556448 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:56:45,215::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:56:45,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670513 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:46,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:46,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912602 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:48,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:49,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912065 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:51,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:51,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.534005 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:56:52,062::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:56:52,063::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:56:53,327::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:56:53,338::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643195 s, 577 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:53,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:53,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912614 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:56:55,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:56:55,241::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653487 s, 800 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:56:55,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:56,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548925 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:56:56,555::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 16:56:56,733::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:56:58,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:56:58,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913653 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:00,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:00,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913847 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:57:02,065::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:57:02,066::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:57:02,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:02,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912964 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:57:03,343::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:57:03,360::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000612016 s, 606 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:04,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:05,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555958 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:57:05,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:57:05,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649824 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:07,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:07,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915287 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:09,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:09,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912102 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:11,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:57:11,736::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3ea7929-366f-4b4d-8551-35a0b07b3227`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:57:11,736::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:57:11,998::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:57:11,999::task::1191::Storage.TaskManager.Task::(prepare) Task=`a3ea7929-366f-4b4d-8551-35a0b07b3227`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:57:12,000::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3ea7929-366f-4b4d-8551-35a0b07b3227`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:57:12,000::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:57:12,001::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:57:12,002::task::993::Storage.TaskManager.Task::(_decref) Task=`a3ea7929-366f-4b4d-8551-35a0b07b3227`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 16:57:12,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548468 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:57:12,033::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:57:12,069::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:57:12,070::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:57:13,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:57:13,378::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635697 s, 584 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:14,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:14,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539986 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:57:15,266::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:57:15,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629056 s, 831 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:16,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:17,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37126 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:19,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:20,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091492 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:57:22,080::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:57:22,081::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:57:22,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:22,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914709 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:57:23,381::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:57:23,393::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664735 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:24,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:24,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912489 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:57:25,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:57:25,294::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068872 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:26,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 16:57:26,589::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:57:26,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543506 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:57:27,046::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:57:28,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:29,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913058 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:31,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:31,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915935 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:57:32,092::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:57:32,093::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:57:33,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:33,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914117 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:57:33,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:57:33,408::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067413 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:35,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:57:35,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:57:35,372::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006253 s, 836 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:35,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.109284 s, 9.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:37,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:37,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915496 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:39,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:39,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913446 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:41,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:41,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920347 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:57:42,058::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:57:42,103::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:57:42,104::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:57:43,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:57:43,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687394 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:43,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:43,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913431 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:57:45,383::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:57:45,394::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000729888 s, 717 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:45,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:45,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912992 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:48,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:48,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539595 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:50,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:50,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0937853 s, 10.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:57:52,111::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:57:52,112::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:57:52,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:52,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913682 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:57:53,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:57:53,438::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664423 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:54,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:57:54,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913434 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:57:55,398::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:57:55,409::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692333 s, 755 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:57:56,620::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:57:56,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:57:57,072::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:57:57,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502307 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:57:59,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:00,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543224 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:02,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:58:02,124::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:58:02,125::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:58:02,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913803 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:58:03,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:58:03,453::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068146 s, 544 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:04,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:04,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913425 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:58:05,412::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:58:05,426::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 16:58:05,428::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 16:58:05,430::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 16:58:05,432::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 16:58:05,432::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 16:58:05,432::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 16:58:05,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:58:05,445::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673884 s, 776 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:06,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:06,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914424 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:08,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:08,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911891 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:10,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:11,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555426 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:58:12,076::task::595::Storage.TaskManager.Task::(_updateState) Task=`d7f384dc-f362-477f-985e-07defb144ae0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:58:12,076::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:58:12,079::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:58:12,079::task::1191::Storage.TaskManager.Task::(prepare) Task=`d7f384dc-f362-477f-985e-07defb144ae0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:58:12,080::task::595::Storage.TaskManager.Task::(_updateState) Task=`d7f384dc-f362-477f-985e-07defb144ae0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:58:12,080::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:58:12,080::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:58:12,080::task::993::Storage.TaskManager.Task::(_decref) Task=`d7f384dc-f362-477f-985e-07defb144ae0`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:58:12,108::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:58:12,136::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:58:12,137::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:58:13,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 16:58:13,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:58:13,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000602331 s, 616 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:13,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549373 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:58:15,449::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:58:15,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639604 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:15,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:16,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548433 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:18,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:18,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912513 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:20,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:20,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091296 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:58:22,152::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:58:22,154::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:58:22,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:22,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913371 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:58:23,696::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:58:23,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065811 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:24,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:24,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913506 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:58:25,465::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:58:25,477::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000724724 s, 722 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:58:26,653::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:58:26,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:26,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913081 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:58:27,121::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:58:28,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:29,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913556 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:31,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:31,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912907 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:58:32,164::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:58:32,166::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:58:33,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:33,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912747 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:58:33,714::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:58:33,727::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 16:58:33,729::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 16:58:33,732::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 16:58:33,733::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 16:58:33,733::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 16:58:33,734::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 16:58:33,735::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:58:33,751::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637541 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:35,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:35,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913074 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:58:35,481::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:58:35,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068001 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:37,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:37,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914212 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:39,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:39,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910903 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:41,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:41,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913004 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:58:42,135::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:58:42,172::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:58:42,173::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:58:43,756::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:58:43,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686537 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:43,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:43,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921701 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:58:45,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:58:45,595::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00508979 s, 103 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:45,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:46,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.564452 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:48,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:48,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914496 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:50,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:50,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913385 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:58:52,184::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:58:52,185::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:58:52,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:52,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913666 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:58:53,771::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:58:53,782::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674627 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:58:54,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:55,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915056 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:58:55,603::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:58:55,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660287 s, 792 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:58:56,686::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:58:57,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:57,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0934352 s, 11.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:58:57,153::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:58:59,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:58:59,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542506 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:01,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 16:59:02,188::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:59:02,189::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:59:03,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36212 s, 752 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:59:03,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:59:03,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699076 s, 531 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:05,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:05,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912944 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:59:05,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:59:05,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070153 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:07,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:07,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540327 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:09,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:09,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914381 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:11,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:12,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912619 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:59:12,158::task::595::Storage.TaskManager.Task::(_updateState) Task=`f43de36b-de61-4aea-bf32-ce72ebea64a8`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 16:59:12,158::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 16:59:12,161::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:59:12,161::task::1191::Storage.TaskManager.Task::(prepare) Task=`f43de36b-de61-4aea-bf32-ce72ebea64a8`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 16:59:12,161::task::595::Storage.TaskManager.Task::(_updateState) Task=`f43de36b-de61-4aea-bf32-ce72ebea64a8`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 16:59:12,161::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 16:59:12,162::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 16:59:12,162::task::993::Storage.TaskManager.Task::(_decref) Task=`f43de36b-de61-4aea-bf32-ce72ebea64a8`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 16:59:12,189::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:59:12,199::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:59:12,200::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:59:13,801::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:59:13,813::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654946 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:14,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:14,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.520227 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:59:15,636::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:59:15,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00479441 s, 109 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:16,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:16,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913607 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:18,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:18,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915174 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:20,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:20,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914025 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:59:22,211::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:59:22,212::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:59:22,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:23,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913314 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:59:23,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:59:23,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639694 s, 580 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:25,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:25,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091371 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:59:25,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:59:25,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675572 s, 774 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:59:26,718::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:59:27,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 16:59:27,203::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:59:27,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915033 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:29,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:29,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913379 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:31,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:31,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915429 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:59:32,218::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:59:32,219::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:59:33,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:33,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910935 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:59:33,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:59:33,843::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067115 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:35,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 16:59:35,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:35,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104448 s, 9.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:59:35,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00627097 s, 83.4 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:37,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:37,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927133 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:39,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:39,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912244 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:42,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:42,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914081 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:59:42,216::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 16:59:42,230::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:59:42,231::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 16:59:43,847::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:59:43,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066226 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:44,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:44,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914503 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:59:45,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:59:45,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669165 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:46,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:46,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100354 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:48,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:48,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914585 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:50,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:50,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912328 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 16:59:52,243::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 16:59:52,244::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 16:59:52,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:52,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914353 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 16:59:53,862::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 16:59:53,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656042 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 16:59:54,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:54,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925596 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 16:59:55,790::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 16:59:55,801::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707656 s, 739 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 16:59:56,754::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 16:59:56,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:57,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914327 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 16:59:57,229::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 16:59:59,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 16:59:59,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914409 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:01,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:01,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913366 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:00:02,255::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:00:02,256::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:00:03,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:03,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915358 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:00:03,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:00:03,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00106316 s, 349 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:05,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:05,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913415 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:00:05,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:00:05,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662702 s, 789 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:07,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:07,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910463 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:09,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:09,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913107 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:11,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:11,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911828 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:00:12,232::task::595::Storage.TaskManager.Task::(_updateState) Task=`1ba5d78a-2221-4f2d-b44b-69b95a8fb7c8`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:00:12,233::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:00:12,236::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:00:12,236::task::1191::Storage.TaskManager.Task::(prepare) Task=`1ba5d78a-2221-4f2d-b44b-69b95a8fb7c8`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:00:12,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`1ba5d78a-2221-4f2d-b44b-69b95a8fb7c8`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:00:12,236::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:00:12,236::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:00:12,237::task::993::Storage.TaskManager.Task::(_decref) Task=`1ba5d78a-2221-4f2d-b44b-69b95a8fb7c8`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:00:12,264::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 17:00:12,268::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:00:12,269::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:00:13,892::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:00:13,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650608 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:13,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:14,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913773 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:00:15,822::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:00:15,837::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00077606 s, 674 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:16,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:16,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968422 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:18,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:18,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914588 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:20,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:20,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913743 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:00:22,276::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:00:22,277::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:00:22,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:22,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091255 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:00:23,907::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:00:23,919::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658753 s, 563 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:24,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:24,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913332 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:00:25,842::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:00:25,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698409 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:26,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:00:26,786::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:00:26,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912714 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:00:27,278::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:00:28,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:28,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:30,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:31,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543816 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:00:32,292::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:00:32,293::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:00:33,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:33,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913818 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:00:33,923::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:00:33,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659757 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:35,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:35,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091342 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:00:35,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:00:35,870::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663375 s, 788 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:37,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:37,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914272 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:39,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:39,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917527 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:42,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:42,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913491 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:00:42,290::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >GuestMonitor-vm11::ERROR::2014-07-14 17:00:42,300::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:00:42,302::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:00:43,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:00:43,949::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000722964 s, 513 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:44,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:44,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964192 s, 10.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:00:45,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:00:45,889::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000724751 s, 722 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:46,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:46,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0951354 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:48,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:48,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0937109 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:50,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:50,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912547 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:00:52,311::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:00:52,312::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:00:52,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:52,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912938 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:00:53,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:00:53,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069679 s, 532 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:00:54,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:54,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914383 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:00:55,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:00:55,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067823 s, 771 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:00:56,818::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:00:56,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:57,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914963 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:00:57,304::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:00:59,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:00:59,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55004 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:01,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:01,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091372 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:01:02,323::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:01:02,324::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:01:03,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:01:04,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:01:04,351::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670927 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:04,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.564441 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:01:05,914::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:01:05,927::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658988 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:06,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:06,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914589 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:08,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:08,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0947867 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:10,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:10,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912167 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:01:12,307::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ab1dec0-2d80-4f49-8946-2eaecbf143bb`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:01:12,307::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:01:12,310::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:01:12,311::task::1191::Storage.TaskManager.Task::(prepare) Task=`5ab1dec0-2d80-4f49-8946-2eaecbf143bb`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:01:12,311::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ab1dec0-2d80-4f49-8946-2eaecbf143bb`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:01:12,311::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:01:12,311::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:01:12,311::task::993::Storage.TaskManager.Task::(_decref) Task=`5ab1dec0-2d80-4f49-8946-2eaecbf143bb`::ref 0 aborting False >GuestMonitor-vm11::ERROR::2014-07-14 17:01:12,334::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:01:12,337::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:01:12,344::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:01:12,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:12,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912441 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:01:14,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:01:14,371::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649481 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:14,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:14,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091515 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:01:15,933::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:01:15,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00248798 s, 210 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:16,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:17,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091365 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:19,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:19,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913551 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:21,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:21,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912349 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:01:22,347::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:01:22,348::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:01:23,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:23,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558246 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:01:24,374::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:01:24,386::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635219 s, 584 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:25,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:01:26,465::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:01:26,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643772 s, 812 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:26,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.578478 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:01:26,850::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:01:27,359::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:01:28,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:28,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912395 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:30,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:30,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091438 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:01:32,355::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:01:32,357::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:01:32,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:32,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911877 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:01:34,389::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:01:34,401::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682368 s, 544 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:34,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:34,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912715 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:01:36,491::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:01:36,503::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649099 s, 806 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:36,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:37,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914545 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:39,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:39,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916334 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:41,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:41,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.460374 s, 2.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:01:42,368::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:01:42,370::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:01:42,375::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:01:43,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:43,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912711 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:01:44,405::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:01:44,416::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690708 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:45,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:46,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.538004 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:01:46,509::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:01:46,522::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000790847 s, 661 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:48,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:48,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918159 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:50,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:50,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918005 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:01:52,381::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:01:52,382::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:01:52,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:53,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.508321 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:01:54,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:01:54,431::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680242 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:55,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:55,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923683 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:01:56,530::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:01:56,541::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685174 s, 763 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:01:56,888::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:01:57,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:01:57,400::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:01:57,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914259 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:01:59,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:01:59,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912537 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:01,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:02,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55755 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:02:02,398::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:02:02,400::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:02:04,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:04,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912221 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:02:04,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:02:04,446::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680678 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:06,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:06,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914266 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:02:06,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:02:06,556::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692775 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:08,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:08,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913661 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:10,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:11,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542859 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:02:12,403::task::595::Storage.TaskManager.Task::(_updateState) Task=`cebb197a-4633-493f-8ce1-49983a829d45`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:02:12,403::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:02:12,406::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:02:12,406::task::1191::Storage.TaskManager.Task::(prepare) Task=`cebb197a-4633-493f-8ce1-49983a829d45`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:02:12,406::task::595::Storage.TaskManager.Task::(_updateState) Task=`cebb197a-4633-493f-8ce1-49983a829d45`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:02:12,406::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:02:12,407::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:02:12,407::task::993::Storage.TaskManager.Task::(_decref) Task=`cebb197a-4633-493f-8ce1-49983a829d45`::ref 0 aborting False >GuestMonitor-vm11::ERROR::2014-07-14 17:02:12,410::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:02:12,412::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:02:12,436::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:02:13,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:13,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916046 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:02:14,450::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:02:14,462::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637457 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:15,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:15,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915198 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:02:16,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:02:16,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065976 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:17,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:17,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910369 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:19,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:19,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913769 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:21,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:21,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913103 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:02:22,419::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:02:22,420::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:02:23,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:23,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916167 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:02:24,466::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:02:24,478::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657755 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:25,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:25,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915003 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:02:26,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:02:26,588::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692567 s, 755 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:02:26,922::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:02:27,453::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:02:27,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:28,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557834 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:30,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:30,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912757 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:02:32,431::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:02:32,432::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:02:32,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:32,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924696 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:02:34,481::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:02:34,492::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658268 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:34,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:34,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914131 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:02:36,592::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:02:36,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000634611 s, 824 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:36,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:36,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912657 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:39,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:39,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914726 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:41,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:41,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.506277 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:02:42,444::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:02:42,445::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:02:42,466::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:02:43,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:43,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913775 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:02:44,496::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:02:44,508::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653044 s, 568 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:45,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:45,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912203 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:02:46,608::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:02:46,620::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000701433 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:47,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:48,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916136 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:50,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:50,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913695 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:52,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:52,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915654 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:02:52,454::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:02:52,455::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:02:54,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:54,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915423 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:02:54,511::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:02:54,523::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684525 s, 542 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:02:56,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:56,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913691 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:02:56,624::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:02:56,635::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681039 s, 768 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:02:56,954::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:02:57,479::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:02:58,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:02:58,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914044 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:00,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:01,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.53552 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:03:02,462::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:03:02,463::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:03:03,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:03,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914873 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:03:05,012::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:05,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:03:05,553::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00182721 s, 203 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:05,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927826 s, 11.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:03:06,637::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:03:06,650::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:03:06,652::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:03:06,655::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:03:06,656::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:03:06,656::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:03:06,657::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:03:06,658::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:03:06,669::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678684 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:07,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:07,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091022 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:09,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:09,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917911 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:11,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:12,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540145 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:03:12,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:03:12,475::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:03:12,482::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d44bf19-4d7b-457b-a9d1-c575082282bb`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:03:12,482::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:03:12,485::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:03:12,485::task::1191::Storage.TaskManager.Task::(prepare) Task=`0d44bf19-4d7b-457b-a9d1-c575082282bb`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:03:12,485::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d44bf19-4d7b-457b-a9d1-c575082282bb`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:03:12,485::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:03:12,485::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:03:12,485::task::993::Storage.TaskManager.Task::(_decref) Task=`0d44bf19-4d7b-457b-a9d1-c575082282bb`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:03:12,513::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:03:14,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:14,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914032 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:03:15,645::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:03:15,659::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000614849 s, 603 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:16,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:03:16,680::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:03:16,692::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669114 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:16,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10849 s, 9.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:18,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:18,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920018 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:20,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:20,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912701 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:03:22,484::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:03:22,485::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:03:22,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:23,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912792 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:25,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:25,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910606 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:03:25,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:03:25,676::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670498 s, 553 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:03:26,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:03:26,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690094 s, 758 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:03:26,986::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:03:27,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:03:27,526::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:03:27,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507464 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:29,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:30,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543737 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:32,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:32,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913095 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:03:32,496::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:03:32,497::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:03:34,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:34,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912222 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:03:35,678::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:03:35,691::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:03:35,693::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:03:35,696::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:03:35,697::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:03:35,698::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:03:35,698::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:03:35,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:03:35,710::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670449 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:36,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:36,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914754 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:03:36,716::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:03:36,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659683 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:38,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:39,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544801 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:41,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:41,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912454 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:03:42,505::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:03:42,506::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:03:42,538::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:03:43,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:43,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912988 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:45,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:45,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999396 s, 10.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:03:45,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:03:45,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681219 s, 545 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:03:46,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:03:46,750::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651405 s, 803 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:47,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:47,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912999 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:49,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:49,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913916 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:52,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:52,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914161 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:03:52,517::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:03:52,519::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:03:54,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:54,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091335 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:03:55,762::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:03:55,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691219 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:03:56,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:56,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913497 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:03:56,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:03:56,765::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698966 s, 748 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:03:57,019::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:03:57,551::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:03:58,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:03:58,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913283 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:00,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:00,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912295 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:04:02,529::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:04:02,531::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:04:02,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:02,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914093 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:04,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:04,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913814 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:04:05,778::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:04:05,789::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697875 s, 532 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:06,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:04:07,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:04:08,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.541828 s, 1.0 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:08,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.637086 s, 1.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:10,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:10,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913189 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:12,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:12,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913331 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:04:12,542::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:04:12,543::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:04:12,554::task::595::Storage.TaskManager.Task::(_updateState) Task=`6e75a37a-e9c7-410b-84cb-ab78353d45fe`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:04:12,554::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:04:12,557::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:04:12,557::task::1191::Storage.TaskManager.Task::(prepare) Task=`6e75a37a-e9c7-410b-84cb-ab78353d45fe`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:04:12,557::task::595::Storage.TaskManager.Task::(_updateState) Task=`6e75a37a-e9c7-410b-84cb-ab78353d45fe`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:04:12,557::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:04:12,557::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:04:12,558::task::993::Storage.TaskManager.Task::(_decref) Task=`6e75a37a-e9c7-410b-84cb-ab78353d45fe`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:04:12,585::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:04:14,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:15,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543198 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:04:15,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:04:15,808::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000609633 s, 609 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:17,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:17,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913151 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:04:18,224::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:04:18,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643953 s, 812 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:19,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:19,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091326 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:21,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:21,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912672 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:04:22,555::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:04:22,556::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:04:23,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:23,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5472 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:04:25,814::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:04:25,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650256 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:25,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:26,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912903 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:04:27,051::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:04:27,597::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:04:28,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:28,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914022 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:04:28,274::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:04:28,286::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679956 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:30,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:30,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912982 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:32,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:04:32,563::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:04:32,565::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:04:32,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541092 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:34,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:34,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0932595 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:04:35,829::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:04:35,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685646 s, 541 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:37,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:37,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914555 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:04:38,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:04:38,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683291 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:39,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:39,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913195 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:41,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:41,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913281 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:04:42,574::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:04:42,575::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:04:42,610::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:04:43,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:43,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913466 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:45,471::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:45,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100014 s, 10.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:04:45,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:04:45,859::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000711953 s, 521 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:47,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:47,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913325 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:04:48,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:04:48,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000714053 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:49,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:49,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913269 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:51,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:52,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913017 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:04:52,585::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:04:52,586::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:04:54,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:54,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55143 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:04:55,867::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:04:55,893::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637507 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:56,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:56,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913691 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:04:57,091::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:04:57,622::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:04:58,322::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:04:58,333::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649295 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:04:58,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:04:58,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912987 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:00,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:01,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515191 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:05:02,597::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:05:02,598::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:05:03,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:03,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091293 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:05,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:05:06,063::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:05:06,076::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662073 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:06,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.523711 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:08,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:08,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913113 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:05:08,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:05:08,348::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673715 s, 776 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:10,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:10,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914151 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:12,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:12,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913895 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:05:12,608::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:05:12,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:05:12,625::task::595::Storage.TaskManager.Task::(_updateState) Task=`562b824c-a4d9-405c-979c-729a002f0457`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:05:12,625::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:05:12,628::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:05:12,628::task::1191::Storage.TaskManager.Task::(prepare) Task=`562b824c-a4d9-405c-979c-729a002f0457`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:05:12,628::task::595::Storage.TaskManager.Task::(_updateState) Task=`562b824c-a4d9-405c-979c-729a002f0457`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:05:12,629::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:05:12,629::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:05:12,629::task::993::Storage.TaskManager.Task::(_decref) Task=`562b824c-a4d9-405c-979c-729a002f0457`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:05:12,657::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:05:14,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:14,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913884 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:05:16,088::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:05:16,103::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00061468 s, 604 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:16,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:16,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913225 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:05:18,352::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:05:18,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652747 s, 801 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:18,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:19,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549499 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:21,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:21,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544932 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:05:22,619::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:05:22,620::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:05:23,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:23,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911807 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:25,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:26,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917862 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:05:26,111::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:05:26,123::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664569 s, 558 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:05:27,127::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:05:27,669::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:05:28,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:28,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910037 s, 11.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:05:28,368::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:05:28,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673006 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:30,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:30,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914731 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:32,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:05:32,631::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:05:32,632::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:05:32,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544925 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:34,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:34,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912773 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:05:36,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:05:36,139::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000728465 s, 509 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:36,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:37,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924456 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:05:38,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:05:38,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666185 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:39,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:39,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912047 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:41,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:41,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913553 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:05:42,643::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:05:42,644::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:05:42,682::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:05:43,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:43,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912876 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:45,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:45,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.513243 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:05:46,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:05:46,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00483661 s, 76.7 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:48,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:05:48,663::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:05:48,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000778146 s, 672 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:48,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.537346 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:50,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:50,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.267722 s, 3.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:05:52,655::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:05:52,656::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:05:52,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:53,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.517267 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:55,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:55,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913656 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:05:56,280::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:05:56,292::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691791 s, 536 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:05:57,168::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:05:57,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:05:57,696::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:05:57,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914037 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:05:58,685::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:05:58,698::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687156 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:05:59,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:05:59,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912649 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:01,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:02,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542623 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:06:02,669::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:06:02,670::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:06:04,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:04,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914389 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:06:06,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:06:06,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682287 s, 544 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:06,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:06,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914539 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:08,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:06:08,703::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:06:08,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630368 s, 830 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:08,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0974132 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:10,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:10,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913314 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:06:12,681::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:06:12,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:06:12,701::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcac5d73-aac4-463f-b73c-6050dbdf19ff`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:06:12,701::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:06:12,703::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:06:12,704::task::1191::Storage.TaskManager.Task::(prepare) Task=`fcac5d73-aac4-463f-b73c-6050dbdf19ff`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:06:12,704::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcac5d73-aac4-463f-b73c-6050dbdf19ff`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:06:12,704::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:06:12,704::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:06:12,704::task::993::Storage.TaskManager.Task::(_decref) Task=`fcac5d73-aac4-463f-b73c-6050dbdf19ff`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:06:12,730::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:06:12,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:13,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912609 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:15,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:15,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09142 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:06:16,312::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:06:16,326::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000611354 s, 607 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:17,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:17,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091192 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:06:18,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:06:18,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000625516 s, 836 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:19,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:19,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913289 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:21,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:21,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913487 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:06:22,693::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:06:22,694::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:06:23,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:23,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919696 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:25,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:25,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911865 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:06:26,332::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:06:26,344::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663499 s, 559 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:06:27,200::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:06:27,749::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:06:27,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:27,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913054 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:06:28,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:06:28,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665487 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:29,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:30,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544365 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:32,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:32,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913127 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:06:32,701::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:06:32,703::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:06:34,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:34,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914386 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:06:36,347::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:06:36,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683712 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:36,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:37,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542642 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:06:38,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:06:38,842::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639207 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:39,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:39,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914596 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:41,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:41,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914853 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:06:42,710::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:06:42,713::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:06:42,760::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:06:43,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:43,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913992 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:45,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:45,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0966498 s, 10.6 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:06:46,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:06:46,381::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000695833 s, 533 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:47,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:47,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917194 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:06:48,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:06:48,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664166 s, 787 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:49,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:50,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913896 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:52,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:52,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915801 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:06:52,722::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:06:52,724::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:06:54,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:54,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913981 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:56,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:06:57,232::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 17:06:57,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:57,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37555 s, 744 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:06:57,703::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678043 s, 547 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:06:57,772::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:06:58,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:06:58,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691772 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:06:59,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:06:59,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912153 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:01,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:01,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911834 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:07:02,731::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:07:02,732::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:07:03,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:04,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515108 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:06,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:06,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913265 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:07:07,711::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:07:07,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667441 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:08,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:08,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911677 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:07:08,877::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:07:08,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659075 s, 794 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:10,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:10,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09129 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:07:12,739::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:07:12,741::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:07:12,775::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2391060-8c3f-4ec7-8e14-73405d1f45db`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:07:12,775::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:07:12,778::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:07:12,779::task::1191::Storage.TaskManager.Task::(prepare) Task=`c2391060-8c3f-4ec7-8e14-73405d1f45db`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:07:12,779::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2391060-8c3f-4ec7-8e14-73405d1f45db`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:07:12,779::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:07:12,779::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:07:12,779::task::993::Storage.TaskManager.Task::(_decref) Task=`c2391060-8c3f-4ec7-8e14-73405d1f45db`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:07:12,808::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:07:12,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:12,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912755 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:14,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:15,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543306 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:17,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:17,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919381 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:07:17,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:07:17,737::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651441 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:07:18,892::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:07:18,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656915 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:19,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:19,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914348 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:21,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:21,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091352 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:07:22,751::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:07:22,753::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:07:23,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:23,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914639 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:26,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:26,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544623 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:07:27,268::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 17:07:27,741::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:07:27,752::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636535 s, 583 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:07:27,820::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:07:28,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:28,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912954 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:07:28,908::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:07:28,919::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716248 s, 730 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:30,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:30,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915672 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:07:32,764::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:07:32,765::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:07:32,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:32,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915493 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:34,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:35,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913929 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:37,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:37,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914484 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:07:37,756::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:07:37,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693607 s, 535 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:07:38,924::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:07:38,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668783 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:39,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:39,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54013 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:41,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:41,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914386 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:07:42,778::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:07:42,779::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:07:42,832::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:07:43,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:43,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132978 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:46,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:46,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100168 s, 10.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:07:47,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:07:47,785::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665461 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:48,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:07:49,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:07:49,617::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000708912 s, 738 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:49,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.34874 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:51,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:51,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923296 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:07:52,788::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:07:52,789::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:07:53,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:53,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912562 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:07:55,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:55,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915318 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:07:57,299::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 17:07:57,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:07:57,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645508 s, 575 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:07:57,845::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:07:57,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:07:58,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916114 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:07:59,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:07:59,637::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668092 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:00,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:00,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.53672 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:02,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:02,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913545 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:08:02,803::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:08:02,804::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:08:04,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:04,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912066 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:06,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:07,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915472 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:08:07,803::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:08:07,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000720639 s, 515 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:09,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:09,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549061 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:08:09,639::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:08:09,656::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:08:09,657::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:08:09,660::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:08:09,661::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:08:09,661::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:08:09,662::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:08:09,663::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:08:09,674::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682892 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:11,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:12,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.509572 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:08:12,819::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:08:12,820::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:08:12,848::task::595::Storage.TaskManager.Task::(_updateState) Task=`fd2c3293-c863-4564-8961-0cb49a38f5e9`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:08:12,848::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:08:12,851::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:08:12,851::task::1191::Storage.TaskManager.Task::(prepare) Task=`fd2c3293-c863-4564-8961-0cb49a38f5e9`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:08:12,851::task::595::Storage.TaskManager.Task::(_updateState) Task=`fd2c3293-c863-4564-8961-0cb49a38f5e9`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:08:12,851::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:08:12,851::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:08:12,852::task::993::Storage.TaskManager.Task::(_decref) Task=`fd2c3293-c863-4564-8961-0cb49a38f5e9`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:08:12,878::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:08:14,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:14,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921691 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:16,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:16,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.29199 s, 3.5 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:08:17,927::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:08:17,939::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000616226 s, 602 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:18,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:18,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913946 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:08:19,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:08:19,690::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657268 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:20,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:20,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912626 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:22,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:08:22,850::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:08:22,852::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:08:22,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915784 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:24,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:25,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913446 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:27,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:27,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913693 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:08:27,332::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:08:27,900::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:08:27,943::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:08:27,954::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617435 s, 601 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:29,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:29,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913025 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:08:29,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:08:29,705::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706912 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:31,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:31,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916108 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:08:32,859::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:08:32,860::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:08:33,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:33,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913402 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:35,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:35,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912645 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:37,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:37,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912632 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:08:37,956::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:08:37,969::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:08:37,970::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:08:37,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:08:37,975::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:08:37,975::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:08:37,975::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:08:37,976::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:08:37,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645757 s, 575 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:08:39,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:08:39,721::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658223 s, 795 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:39,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:39,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914927 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:41,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:41,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912948 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:08:42,865::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:08:42,867::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:08:42,912::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:08:43,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:44,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914374 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:46,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:46,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.525374 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:08:48,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:08:48,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687755 s, 539 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:48,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:48,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913895 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:08:49,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:08:49,736::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699364 s, 748 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:50,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:52,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36823 s, 748 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:08:52,877::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:08:52,879::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:08:54,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:54,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912822 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:56,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:56,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910842 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:08:57,367::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:08:57,924::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:08:58,062::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:08:58,073::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662221 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:08:58,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:08:58,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546832 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:08:59,741::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:08:59,753::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068334 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:00,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:02,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36375 s, 751 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:09:02,890::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:09:02,891::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:09:04,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:04,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913142 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:06,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:06,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913291 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:09:08,076::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:09:08,088::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000666798 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:08,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:08,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912007 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:09:09,759::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:09:09,771::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000704395 s, 742 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:10,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:10,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913776 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:12,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:09:12,900::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:09:12,901::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:09:12,927::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c6ba932-b31b-4d20-8abc-7b7c0f0777fd`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:09:12,927::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:09:12,940::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:09:12,940::task::1191::Storage.TaskManager.Task::(prepare) Task=`7c6ba932-b31b-4d20-8abc-7b7c0f0777fd`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:09:12,941::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c6ba932-b31b-4d20-8abc-7b7c0f0777fd`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:09:12,942::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:09:12,942::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:09:12,943::task::993::Storage.TaskManager.Task::(_decref) Task=`7c6ba932-b31b-4d20-8abc-7b7c0f0777fd`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:09:12,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0957355 s, 10.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:09:12,971::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:09:14,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:15,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916074 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:17,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:17,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091204 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:09:18,092::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:09:18,104::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000633474 s, 586 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:19,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:09:20,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:20,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.596632 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:09:20,523::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00201662 s, 259 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:22,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:22,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912114 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:09:22,908::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:09:22,909::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:09:24,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:24,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913232 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:26,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:26,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912555 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:09:27,404::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:09:27,984::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:09:28,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:09:28,119::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663216 s, 559 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:28,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:28,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913156 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:09:30,533::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:09:30,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671899 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:31,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:31,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091216 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:09:32,920::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:09:32,921::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:09:33,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:33,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.516962 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:35,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:35,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913544 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:37,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:37,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914015 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:09:38,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:09:38,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00103451 s, 359 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:39,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:40,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914199 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:09:40,549::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:09:40,560::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661682 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:42,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:42,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914949 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:09:42,929::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:09:42,930::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:09:42,997::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:09:44,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:44,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913826 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:46,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:46,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999993 s, 10.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:09:48,137::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:09:48,149::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682034 s, 544 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:48,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:48,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917863 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:09:50,564::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:09:50,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000754429 s, 693 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:50,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:50,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132338 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:52,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:52,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912908 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:09:52,942::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:09:52,943::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:09:54,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:55,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551893 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:09:57,438::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:09:57,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:57,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0932067 s, 11.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:09:58,009::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:09:58,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:09:58,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000700973 s, 529 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:09:59,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:09:59,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913102 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:10:00,580::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:10:00,591::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697198 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:01,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:02,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0944999 s, 10.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:10:02,953::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:10:02,954::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:10:04,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:04,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54428 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:06,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:06,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912599 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:10:08,167::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:10:08,178::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686014 s, 541 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:08,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:08,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912638 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:10:10,595::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:10:10,606::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683833 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:10,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:11,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915092 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:10:12,965::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:10:12,967::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:10:13,012::task::595::Storage.TaskManager.Task::(_updateState) Task=`579a93dc-4891-4443-949e-1a02e18f4644`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:10:13,012::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:10:13,015::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:10:13,015::task::1191::Storage.TaskManager.Task::(prepare) Task=`579a93dc-4891-4443-949e-1a02e18f4644`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:10:13,015::task::595::Storage.TaskManager.Task::(_updateState) Task=`579a93dc-4891-4443-949e-1a02e18f4644`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:10:13,016::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:10:13,016::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:10:13,016::task::993::Storage.TaskManager.Task::(_decref) Task=`579a93dc-4891-4443-949e-1a02e18f4644`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:10:13,043::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:10:13,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:13,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915036 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:15,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:15,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915184 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:17,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:17,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091239 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:10:18,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:10:18,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661529 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:19,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:19,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55112 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:10:20,610::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:10:20,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656798 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:21,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:22,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091222 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:10:22,973::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:10:22,975::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:10:24,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:24,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912217 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:26,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:26,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091644 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:10:27,471::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:10:28,055::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:10:28,197::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:10:28,209::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639736 s, 580 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:28,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:28,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915723 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:30,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:30,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914352 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:10:30,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:10:30,637::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681355 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:32,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:32,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914305 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:10:32,986::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:10:32,987::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:10:34,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:34,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913169 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:36,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:36,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912804 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:10:38,212::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:10:38,224::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674148 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:38,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:39,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091405 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:10:40,641::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:10:40,653::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066121 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:41,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:41,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911878 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:10:42,993::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:10:42,994::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:10:43,068::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:10:43,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:44,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543355 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:46,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:46,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918329 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:10:48,227::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:10:48,239::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067611 s, 549 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:48,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:48,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911991 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:50,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:10:50,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:50,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104838 s, 9.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:10:50,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00559727 s, 93.4 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:52,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:52,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913504 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:10:53,006::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:10:53,007::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:10:54,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:54,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913023 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:56,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:57,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913585 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:10:57,504::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:10:58,085::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:10:58,242::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:10:58,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068918 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:10:59,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:10:59,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914918 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:11:00,695::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:11:00,707::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00063066 s, 829 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:01,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:01,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913704 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:11:03,014::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:11:03,016::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:11:03,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:03,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916408 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:05,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:05,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913101 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:07,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:07,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915259 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:11:08,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:11:08,271::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006915 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:09,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:09,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913741 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:11:10,710::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:11:10,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668124 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:11,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:11,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913159 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:11:13,024::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:11:13,025::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:11:13,088::task::595::Storage.TaskManager.Task::(_updateState) Task=`966f6c1a-a9cb-4712-a037-018ad803adc5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:11:13,088::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:11:13,091::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:11:13,091::task::1191::Storage.TaskManager.Task::(prepare) Task=`966f6c1a-a9cb-4712-a037-018ad803adc5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:11:13,091::task::595::Storage.TaskManager.Task::(_updateState) Task=`966f6c1a-a9cb-4712-a037-018ad803adc5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:11:13,091::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:11:13,092::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:11:13,092::task::993::Storage.TaskManager.Task::(_decref) Task=`966f6c1a-a9cb-4712-a037-018ad803adc5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:11:13,119::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:11:13,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:14,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558335 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:16,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:16,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913422 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:11:18,275::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:11:18,287::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654497 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:18,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:18,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914578 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:20,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:11:20,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:11:20,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690052 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:20,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108958 s, 9.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:22,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:22,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913356 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:11:23,036::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:11:23,037::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:11:24,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:25,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.553302 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:11:27,535::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:11:27,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:27,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914529 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:11:28,131::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:11:28,291::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:11:28,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648245 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:29,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:30,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551768 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:11:30,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:11:30,832::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000619853 s, 844 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:32,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:32,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913528 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:11:33,044::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:11:33,046::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:11:34,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:34,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913052 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:36,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:36,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912992 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:11:38,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:11:38,317::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660931 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:38,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:39,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541954 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:11:40,837::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:11:40,849::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692296 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:41,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:41,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913664 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:11:43,056::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:11:43,057::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:11:43,143::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:11:43,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:43,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913272 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:45,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:45,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968823 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:47,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:48,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913272 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:11:48,321::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:11:48,332::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659369 s, 563 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:50,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:50,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912371 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:11:50,853::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:11:50,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683721 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:52,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:52,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911908 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:11:53,069::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:11:53,070::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:11:54,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:54,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912939 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:56,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:56,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912346 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:11:57,565::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:11:58,154::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:11:58,335::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:11:58,347::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000703834 s, 527 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:11:58,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:11:58,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09227 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:00,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:00,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916529 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:12:00,868::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:12:00,880::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673336 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:02,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:02,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915264 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:12:03,081::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:12:03,082::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:12:04,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:04,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091208 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:06,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:07,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550223 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:12:08,351::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:12:08,363::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682567 s, 544 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:09,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:09,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914945 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:12:10,883::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:12:10,895::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067291 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:11,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:11,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914524 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:12:13,091::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:12:13,092::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:12:13,157::task::595::Storage.TaskManager.Task::(_updateState) Task=`da5d59a9-1db7-47d4-81f9-0b8af334f9fb`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:12:13,158::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:12:13,160::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:12:13,160::task::1191::Storage.TaskManager.Task::(prepare) Task=`da5d59a9-1db7-47d4-81f9-0b8af334f9fb`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:12:13,161::task::595::Storage.TaskManager.Task::(_updateState) Task=`da5d59a9-1db7-47d4-81f9-0b8af334f9fb`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:12:13,161::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:12:13,161::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:12:13,161::task::993::Storage.TaskManager.Task::(_decref) Task=`da5d59a9-1db7-47d4-81f9-0b8af334f9fb`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:12:13,187::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:12:13,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:13,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913832 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:15,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:16,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100296 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:18,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:18,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912327 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:12:18,367::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:12:18,379::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000638491 s, 581 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:20,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:20,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549753 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:12:20,899::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:12:20,911::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650467 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:22,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:22,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914281 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:12:23,103::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:12:23,105::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:12:24,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:25,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914593 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:27,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:27,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910552 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:12:27,596::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:12:28,199::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:12:28,382::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:12:28,394::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067256 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:29,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:29,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909092 s, 11.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:12:30,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:12:30,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000737208 s, 709 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:31,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:31,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556014 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:12:33,116::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:12:33,117::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:12:33,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:34,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0937312 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:36,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:36,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920372 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:38,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:38,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912879 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:12:38,397::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:12:38,409::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673914 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:40,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:40,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549896 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:12:40,930::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:12:40,942::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671171 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:42,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:42,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912623 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:12:43,124::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:12:43,125::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:12:43,210::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:12:44,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:45,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913044 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:47,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:47,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919943 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:12:48,413::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:12:48,424::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654928 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:49,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:49,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913142 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:12:50,946::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:12:50,957::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691707 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:51,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:51,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915535 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:12:53,136::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:12:53,137::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:12:53,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:53,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912808 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:12:55,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:55,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912118 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:12:57,629::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:12:57,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:12:58,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545822 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:12:58,225::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:12:58,428::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:12:58,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682106 s, 544 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:00,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:00,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913406 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:13:00,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:13:00,973::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674852 s, 775 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:02,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:02,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919584 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:13:03,144::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:13:03,145::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:13:04,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:04,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912921 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:06,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:06,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0936397 s, 10.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:13:08,443::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:13:08,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000610214 s, 608 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:08,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:08,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912806 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:10,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:10,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913495 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:13:10,975::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:13:10,989::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:13:10,990::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:13:10,993::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:13:10,994::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:13:10,994::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:13:10,995::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:13:10,996::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:13:11,008::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000752261 s, 695 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:12,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:13:13,154::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:13:13,156::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:13:13,228::task::595::Storage.TaskManager.Task::(_updateState) Task=`34b7c039-3db6-4fc8-80d4-8cbcf3ef1a35`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:13:13,229::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:13:13,467::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:13:13,468::task::1191::Storage.TaskManager.Task::(prepare) Task=`34b7c039-3db6-4fc8-80d4-8cbcf3ef1a35`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:13:13,468::task::595::Storage.TaskManager.Task::(_updateState) Task=`34b7c039-3db6-4fc8-80d4-8cbcf3ef1a35`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:13:13,469::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:13:13,469::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:13:13,470::task::993::Storage.TaskManager.Task::(_decref) Task=`34b7c039-3db6-4fc8-80d4-8cbcf3ef1a35`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:13:13,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515773 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:13:13,502::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:13:15,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:15,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.113011 s, 9.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:17,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:18,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.533655 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:13:18,464::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:13:18,476::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632948 s, 586 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:20,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:20,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912705 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:13:21,012::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:13:21,024::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675249 s, 775 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:22,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:22,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913136 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:13:23,165::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:13:23,166::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:13:24,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:24,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914135 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:26,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:26,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912349 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:13:27,663::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 17:13:28,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:13:28,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00242358 s, 153 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:13:28,517::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:13:28,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:28,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914323 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:30,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:30,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913478 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:13:31,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:13:31,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665147 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:32,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:33,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913145 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:13:33,174::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:13:33,175::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:13:35,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:35,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914727 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:37,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:37,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545294 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:13:38,498::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:13:38,511::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:13:38,513::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:13:38,516::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:13:38,517::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:13:38,517::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:13:38,518::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:13:38,520::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:13:38,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00401914 s, 92.3 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:39,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:39,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914192 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:13:41,044::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:13:41,056::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659712 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:41,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:41,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912675 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:13:43,183::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:13:43,184::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:13:43,529::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:13:43,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:44,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914732 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:46,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:46,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0965929 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:48,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:48,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913242 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:13:48,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:13:48,553::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677626 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:50,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:50,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912209 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:13:51,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:13:51,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000743755 s, 703 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:52,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:52,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911692 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:13:53,195::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:13:53,196::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:13:54,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:55,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554674 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:57,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:57,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912657 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:13:57,696::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:13:58,541::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:13:58,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:13:58,568::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692699 s, 536 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:13:59,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:13:59,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917976 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:14:01,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:14:01,086::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707006 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:01,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:01,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912396 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:14:03,206::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:14:03,207::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:14:03,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:03,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913397 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:05,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:05,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912078 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:07,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:07,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912941 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:14:08,571::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:14:08,583::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670091 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:09,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:10,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925435 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:14:11,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:14:11,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000645032 s, 811 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:12,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:12,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912635 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:14:13,215::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:14:13,217::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:14:13,545::task::595::Storage.TaskManager.Task::(_updateState) Task=`9688b859-734b-4f4d-acad-3934d3dbe883`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:14:13,545::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:14:13,548::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:14:13,548::task::1191::Storage.TaskManager.Task::(prepare) Task=`9688b859-734b-4f4d-acad-3934d3dbe883`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:14:13,548::task::595::Storage.TaskManager.Task::(_updateState) Task=`9688b859-734b-4f4d-acad-3934d3dbe883`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:14:13,548::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:14:13,549::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:14:13,549::task::993::Storage.TaskManager.Task::(_decref) Task=`9688b859-734b-4f4d-acad-3934d3dbe883`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:14:13,575::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:14:14,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:14,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914144 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:16,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:16,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542948 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:14:18,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:14:18,599::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674308 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:18,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:19,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.535044 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:14:21,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:14:21,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635381 s, 823 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:21,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:21,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916144 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:14:23,227::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:14:23,229::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:14:23,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:23,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914509 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:25,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:25,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915359 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:14:27,728::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:14:27,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:27,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914486 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:14:28,587::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:14:28,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:14:28,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637616 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:29,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:30,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543159 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:14:31,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:14:31,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006722 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:32,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:32,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912564 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:14:33,236::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:14:33,237::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:14:34,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:34,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913319 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:36,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:37,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544847 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:14:38,617::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:14:38,628::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000633745 s, 585 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:39,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:40,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558117 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:14:41,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:14:41,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636199 s, 822 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:42,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:42,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913131 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:14:43,245::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:14:43,247::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:14:43,599::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:14:44,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:44,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913551 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:46,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:46,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100172 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:48,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:14:48,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:14:48,677::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006497 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:48,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106074 s, 9.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:50,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:50,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910784 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:14:51,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:14:51,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006821 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:52,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:14:53,258::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:14:53,259::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:14:53,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549622 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:55,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:14:55,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913663 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:14:57,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:14:57,759::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:14:58,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549462 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:14:58,611::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:14:58,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:14:58,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651441 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:00,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:00,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912438 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:15:01,169::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:15:01,181::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680828 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:02,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:02,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914244 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:15:03,266::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:15:03,267::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:15:04,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:04,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912862 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:06,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:07,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.562542 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:15:08,697::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:15:08,709::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689357 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:09,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:09,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914029 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:11,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:15:11,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:11,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104344 s, 9.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:15:11,340::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00105835 s, 494 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:15:13,277::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:15:13,278::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:15:13,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:13,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913128 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:15:13,614::task::595::Storage.TaskManager.Task::(_updateState) Task=`bddcb3a3-9824-4798-8257-bd29bd5df91e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:15:13,615::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:15:13,618::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:15:13,618::task::1191::Storage.TaskManager.Task::(prepare) Task=`bddcb3a3-9824-4798-8257-bd29bd5df91e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:15:13,618::task::595::Storage.TaskManager.Task::(_updateState) Task=`bddcb3a3-9824-4798-8257-bd29bd5df91e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:15:13,618::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:15:13,618::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:15:13,619::task::993::Storage.TaskManager.Task::(_decref) Task=`bddcb3a3-9824-4798-8257-bd29bd5df91e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:15:13,646::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:15:15,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:15,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913966 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:17,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:17,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914369 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:15:18,713::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:15:18,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660802 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:19,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:19,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913432 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:15:21,344::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:15:21,356::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670785 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:21,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:22,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507344 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:15:23,289::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:15:23,291::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:15:24,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:24,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091737 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:26,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:26,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913436 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:15:27,790::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:15:28,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:28,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100174 s, 10.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:15:28,671::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:15:28,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:15:28,740::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668856 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:30,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:30,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915253 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:15:31,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:15:31,371::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000745419 s, 702 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:32,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:32,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911812 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:15:33,297::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:15:33,298::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:15:34,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:35,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911925 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:37,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:37,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914405 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:15:38,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:15:38,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660353 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:39,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:39,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913506 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:41,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:15:41,879::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:41,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554383 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:15:41,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00339106 s, 154 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:15:43,308::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:15:43,310::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:15:43,685::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:15:43,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:44,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913294 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:46,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:46,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912698 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:48,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:48,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913238 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:15:48,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:15:48,769::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680828 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:50,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:50,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913809 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:15:51,903::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:15:51,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690257 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:52,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:52,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911763 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:15:53,317::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:15:53,318::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:15:54,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:54,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913221 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:56,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:56,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920059 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:15:57,823::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:15:58,697::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:15:58,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:15:58,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000624076 s, 594 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:15:58,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:15:59,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519161 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:01,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:01,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912786 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:16:01,919::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:16:01,931::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695562 s, 752 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:16:03,327::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:16:03,328::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:16:03,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:04,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550547 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:06,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:06,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914822 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:08,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:08,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912845 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:16:08,798::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:16:08,809::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644885 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:10,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:10,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912888 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:16:11,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:16:11,946::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691342 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:12,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:12,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912228 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:16:13,340::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:16:13,341::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:16:13,700::task::595::Storage.TaskManager.Task::(_updateState) Task=`dbc1c6a8-e252-45f4-b5fa-d58b43a884a5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:16:13,700::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:16:13,703::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:16:13,703::task::1191::Storage.TaskManager.Task::(prepare) Task=`dbc1c6a8-e252-45f4-b5fa-d58b43a884a5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:16:13,703::task::595::Storage.TaskManager.Task::(_updateState) Task=`dbc1c6a8-e252-45f4-b5fa-d58b43a884a5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:16:13,703::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:16:13,703::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:16:13,703::task::993::Storage.TaskManager.Task::(_decref) Task=`dbc1c6a8-e252-45f4-b5fa-d58b43a884a5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:16:13,729::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:16:14,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:15,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543968 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:17,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:18,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39786 s, 733 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:16:18,813::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:16:18,825::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000612118 s, 606 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:20,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:20,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091267 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:16:21,950::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:16:21,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631219 s, 829 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:22,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:22,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912759 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:16:23,347::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:16:23,348::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:16:24,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:25,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558191 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:27,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:27,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912959 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:16:27,856::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:16:28,741::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:16:28,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:16:28,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646705 s, 574 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:29,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:29,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913013 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:31,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:16:32,190::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:16:32,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683496 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:32,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.578198 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:16:33,360::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:16:33,361::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:16:34,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:34,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914767 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:36,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:36,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911884 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:38,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:16:38,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:38,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10356 s, 9.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:16:38,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00554902 s, 66.9 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:40,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:41,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561346 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:16:42,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:16:42,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674082 s, 776 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:16:43,372::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:16:43,373::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:16:43,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:16:43,753::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:16:44,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550945 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:46,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:46,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548037 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:48,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:16:49,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:16:49,224::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000619074 s, 599 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:49,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.567216 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:51,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:51,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912178 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:16:52,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:16:52,240::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706934 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:53,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:16:53,384::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:16:53,385::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:16:53,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912963 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:16:55,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:55,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911371 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:16:57,888::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:16:57,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:16:57,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913456 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:16:58,766::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:16:59,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:16:59,241::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685577 s, 541 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:00,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:00,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916274 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:02,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:02,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913938 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:17:02,243::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:17:02,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673556 s, 776 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:17:03,396::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:17:03,397::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:17:04,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:04,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091392 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:06,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:06,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914961 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:08,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:08,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912597 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:17:09,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:17:09,256::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665673 s, 557 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:10,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:10,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091324 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:17:12,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:17:12,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000727848 s, 719 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:12,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:12,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912781 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:17:13,408::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:17:13,410::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:17:13,769::task::595::Storage.TaskManager.Task::(_updateState) Task=`48e4e08a-fd02-44f8-bce3-8b240e1beef9`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:17:13,770::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:17:13,773::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:17:13,773::task::1191::Storage.TaskManager.Task::(prepare) Task=`48e4e08a-fd02-44f8-bce3-8b240e1beef9`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:17:13,773::task::595::Storage.TaskManager.Task::(_updateState) Task=`48e4e08a-fd02-44f8-bce3-8b240e1beef9`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:17:13,773::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:17:13,773::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:17:13,774::task::993::Storage.TaskManager.Task::(_decref) Task=`48e4e08a-fd02-44f8-bce3-8b240e1beef9`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:17:13,801::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:17:14,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:14,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913496 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:16,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:17,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912429 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:19,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:19,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912565 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:17:19,260::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:17:19,272::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654789 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:21,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:21,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542374 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:17:22,274::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:17:22,286::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675439 s, 774 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:17:23,412::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:17:23,413::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:17:23,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:23,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913689 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:25,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:25,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912754 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:17:27,922::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:17:27,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:28,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914199 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:17:28,813::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:17:29,275::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:17:29,287::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653959 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:30,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:30,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913825 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:32,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:17:32,790::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:17:32,803::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681517 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:32,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.578362 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:17:33,420::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:17:33,421::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:17:34,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:34,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912517 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:36,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:37,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927968 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:17:39,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:17:39,301::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662795 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:39,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:39,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913772 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:41,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:41,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915237 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:17:42,813::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:17:42,825::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686895 s, 761 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:17:43,429::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:17:43,431::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:17:43,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:17:43,826::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:17:43,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912573 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:45,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:46,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556798 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:48,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:48,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913406 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:17:49,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:17:49,317::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679983 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:50,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:50,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913788 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:52,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:17:52,902::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:52,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104037 s, 9.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:17:52,921::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00644067 s, 81.2 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:17:53,439::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:17:53,440::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:17:54,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:55,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913304 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:17:57,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:57,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913647 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:17:57,956::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:17:58,840::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:17:59,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:17:59,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913735 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:17:59,322::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:17:59,335::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00180232 s, 206 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:01,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:01,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912351 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:18:02,931::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:18:02,943::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000711006 s, 736 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:03,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:18:03,446::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:18:03,447::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:18:03,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911617 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:05,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:05,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912639 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:07,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:07,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911746 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:18:09,338::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:18:09,351::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657011 s, 565 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:09,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:10,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558259 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:12,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:12,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544823 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:18:12,946::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:18:12,959::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:18:12,960::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:18:12,963::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:18:12,964::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:18:12,965::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:18:12,965::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:18:12,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:18:12,977::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062462 s, 837 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:18:13,457::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:18:13,459::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:18:13,843::task::595::Storage.TaskManager.Task::(_updateState) Task=`cded010a-1e15-4ad5-8194-550f39c6cd54`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:18:13,843::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:18:13,846::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:18:13,846::task::1191::Storage.TaskManager.Task::(prepare) Task=`cded010a-1e15-4ad5-8194-550f39c6cd54`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:18:13,846::task::595::Storage.TaskManager.Task::(_updateState) Task=`cded010a-1e15-4ad5-8194-550f39c6cd54`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:18:13,847::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:18:13,847::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:18:13,847::task::993::Storage.TaskManager.Task::(_decref) Task=`cded010a-1e15-4ad5-8194-550f39c6cd54`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:18:13,874::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:18:14,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:15,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913124 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:17,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:17,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911773 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:19,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:19,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0939706 s, 10.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:18:19,357::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:18:19,369::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657337 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:21,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:21,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911852 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:18:22,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:18:22,993::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657102 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:23,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:18:23,473::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:18:23,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:18:23,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911979 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:25,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:25,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091242 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:27,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:27,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912311 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:18:27,997::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:18:28,886::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:18:29,372::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:18:29,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677092 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:29,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:30,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561329 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:32,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:32,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912253 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:18:32,997::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:18:33,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000732642 s, 714 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:18:33,485::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:18:33,486::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:18:34,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:34,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913609 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:36,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:36,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091545 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:38,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:39,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544219 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:18:39,386::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:18:39,399::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:18:39,400::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:18:39,403::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:18:39,405::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:18:39,405::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:18:39,405::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:18:39,406::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:18:39,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652572 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:41,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:41,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556003 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:18:43,014::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:18:43,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659866 s, 793 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:18:43,492::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:18:43,494::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:18:43,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:18:43,898::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:18:43,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913065 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:45,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:46,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102245 s, 10.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:48,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:48,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091196 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:18:49,422::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:18:49,433::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00064662 s, 574 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:50,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:50,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554668 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:52,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:18:53,047::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:18:53,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00123599 s, 423 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:53,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103872 s, 9.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:18:53,500::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:18:53,502::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:18:55,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:55,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913038 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:18:57,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:57,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913281 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:18:58,039::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:18:58,911::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:18:59,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:18:59,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:18:59,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.601003 s, 1.7 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:19:00,002::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675643 s, 549 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:01,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:02,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557955 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:19:03,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:19:03,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000658012 s, 795 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:19:03,513::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:19:03,514::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:19:04,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:04,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091357 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:06,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:06,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913525 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:08,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:08,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914339 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:19:10,006::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:19:10,018::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680116 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:10,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:10,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915486 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:12,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:19:13,099::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:13,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103794 s, 9.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:19:13,115::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00274693 s, 190 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:19:13,517::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:19:13,519::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:19:13,914::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb9b0360-601d-456f-8543-f3251a618898`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:19:13,915::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:19:13,917::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:19:13,918::task::1191::Storage.TaskManager.Task::(prepare) Task=`bb9b0360-601d-456f-8543-f3251a618898`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:19:13,918::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb9b0360-601d-456f-8543-f3251a618898`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:19:13,918::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:19:13,919::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:19:13,919::task::993::Storage.TaskManager.Task::(_decref) Task=`bb9b0360-601d-456f-8543-f3251a618898`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:19:13,945::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:19:15,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:15,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912923 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:17,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:17,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912934 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:19,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:19,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912437 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:19:20,022::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:19:20,034::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066789 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:21,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:21,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915165 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:19:23,124::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:19:23,136::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067299 s, 777 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:19:23,530::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:19:23,531::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:19:23,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:24,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541589 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:26,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:26,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091672 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:19:28,071::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:19:28,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:28,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913114 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:19:28,958::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:19:30,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:19:30,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646241 s, 574 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:30,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:30,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912061 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:32,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:32,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912022 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:19:33,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:19:33,151::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695575 s, 752 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:19:33,540::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:19:33,541::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:19:34,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:34,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912909 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:36,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:36,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912949 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:38,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:38,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913013 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:19:40,053::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:19:40,064::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665893 s, 557 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:40,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:41,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542733 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:19:43,155::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:19:43,167::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626813 s, 834 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:19:43,552::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:19:43,553::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:19:43,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:19:43,969::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:19:44,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558045 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:46,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:46,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100269 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:48,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:49,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38595 s, 739 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:19:50,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:19:50,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629251 s, 590 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:51,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:52,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544312 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:19:53,171::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:19:53,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706552 s, 740 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:19:53,562::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:19:53,563::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:19:54,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:54,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551835 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:19:56,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:57,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911623 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:19:58,104::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:19:58,982::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:19:59,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:19:59,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560073 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:20:00,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:20:00,095::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000711454 s, 521 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:01,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:01,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091195 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:20:03,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:20:03,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683238 s, 765 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:20:03,575::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:20:03,576::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:20:03,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:03,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911866 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:05,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:06,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091383 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:08,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:09,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39634 s, 733 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:20:10,098::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:20:10,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662275 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:11,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:12,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546212 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:20:13,202::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:20:13,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000710453 s, 736 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:20:13,587::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:20:13,588::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:20:13,984::task::595::Storage.TaskManager.Task::(_updateState) Task=`cdb7fe04-18ba-4a4d-8567-3bd853a98245`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:20:13,985::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:20:13,987::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:20:13,988::task::1191::Storage.TaskManager.Task::(prepare) Task=`cdb7fe04-18ba-4a4d-8567-3bd853a98245`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:20:13,988::task::595::Storage.TaskManager.Task::(_updateState) Task=`cdb7fe04-18ba-4a4d-8567-3bd853a98245`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:20:13,988::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:20:13,988::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:20:13,988::task::993::Storage.TaskManager.Task::(_decref) Task=`cdb7fe04-18ba-4a4d-8567-3bd853a98245`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:20:14,016::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:20:14,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:14,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550422 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:16,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:16,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913182 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:18,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:18,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912185 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:20:20,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:20:20,126::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656518 s, 565 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:20,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:20,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912458 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:22,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:23,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091382 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:20:23,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:20:23,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655869 s, 797 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:20:23,596::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:20:23,597::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:20:25,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:25,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913595 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:27,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:27,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912071 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:20:28,135::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:20:29,030::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:20:29,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:29,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912208 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:20:30,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:20:30,141::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667899 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:31,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:31,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542299 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:20:33,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:20:33,244::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000624945 s, 837 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:20:33,610::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:20:33,612::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:20:33,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:34,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552037 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:36,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:36,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912154 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:38,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:38,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913084 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:20:40,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:20:40,156::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656509 s, 565 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:40,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:41,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550027 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:20:43,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:20:43,260::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065417 s, 799 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:43,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:43,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912564 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:20:43,623::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:20:43,624::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:20:44,042::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:20:45,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:45,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920033 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:47,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:48,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:50,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:20:50,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:20:50,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667701 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:50,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106841 s, 9.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:52,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:52,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558249 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:20:53,264::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:20:53,276::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000725054 s, 721 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:20:53,633::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:20:53,634::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:20:54,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:54,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097725 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:20:56,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:57,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915704 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:20:58,174::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:20:59,055::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:20:59,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:20:59,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913043 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:21:00,240::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:21:00,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651297 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:01,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:01,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091381 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:21:03,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:21:03,291::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656491 s, 797 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:03,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:03,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912638 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:21:03,640::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:21:03,642::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:21:05,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:05,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911986 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:07,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:07,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912793 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:09,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:09,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912499 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:21:10,254::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:21:10,266::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690942 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:11,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:11,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923139 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:21:13,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:21:13,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690293 s, 758 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:21:13,649::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:21:13,651::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:21:13,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:21:14,058::task::595::Storage.TaskManager.Task::(_updateState) Task=`f2351b54-282e-4075-bb51-621585bb5af7`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:21:14,059::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:21:14,458::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:21:14,459::task::1191::Storage.TaskManager.Task::(prepare) Task=`f2351b54-282e-4075-bb51-621585bb5af7`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:21:14,460::task::595::Storage.TaskManager.Task::(_updateState) Task=`f2351b54-282e-4075-bb51-621585bb5af7`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:21:14,460::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:21:14,461::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:21:14,462::task::993::Storage.TaskManager.Task::(_decref) Task=`f2351b54-282e-4075-bb51-621585bb5af7`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:21:14,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54693 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:21:14,493::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:21:16,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:17,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550665 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:19,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:19,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912365 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:21:20,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:21:20,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626302 s, 592 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:21,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:21,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912311 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:23,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:21:23,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:21:23,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698166 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:23,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.111398 s, 9.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:21:23,659::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:21:23,660::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:21:25,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:25,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912298 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:27,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:27,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091394 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:21:28,207::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:21:29,506::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:21:29,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:29,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912242 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:21:30,286::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:21:30,307::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000610374 s, 608 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:31,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:31,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913864 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:21:33,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:21:33,429::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000740516 s, 706 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:21:33,672::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:21:33,673::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:21:33,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:34,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913464 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:36,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:36,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915932 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:38,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:38,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:40,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:21:40,824::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:40,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.572402 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:21:40,841::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000749214 s, 495 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:42,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:42,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912311 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:21:43,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:21:43,445::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000638196 s, 819 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:21:43,684::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:21:43,685::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:21:44,517::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:21:44,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:45,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.226515 s, 4.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:47,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:47,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545816 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:49,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:49,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912745 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:21:50,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:21:50,857::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677279 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:51,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:52,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558361 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:21:53,449::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:21:53,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676497 s, 773 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:21:53,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:21:53,697::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:21:54,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:54,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911841 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:21:56,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:56,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912908 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:21:58,246::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:21:58,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:21:58,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912786 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:21:59,529::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:22:00,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:22:00,860::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:22:01,399::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0011576 s, 320 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:01,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543471 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:03,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:22:03,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:22:03,531::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000725551 s, 721 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:03,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106984 s, 9.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:22:03,708::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:22:03,709::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:22:05,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:05,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913199 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:07,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:07,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913517 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:09,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:10,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547799 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:22:11,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:22:11,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000987761 s, 376 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:12,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:12,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542393 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:22:13,537::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:22:13,549::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716488 s, 730 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:22:13,721::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:22:13,722::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:22:14,532::task::595::Storage.TaskManager.Task::(_updateState) Task=`e5ffdb60-0ad1-4f93-8c14-a657d79dbaba`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:22:14,532::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:22:14,535::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:22:14,535::task::1191::Storage.TaskManager.Task::(prepare) Task=`e5ffdb60-0ad1-4f93-8c14-a657d79dbaba`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:22:14,535::task::595::Storage.TaskManager.Task::(_updateState) Task=`e5ffdb60-0ad1-4f93-8c14-a657d79dbaba`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:22:14,535::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:22:14,535::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:22:14,535::task::993::Storage.TaskManager.Task::(_decref) Task=`e5ffdb60-0ad1-4f93-8c14-a657d79dbaba`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:22:14,561::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:22:14,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:15,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913141 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:17,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:17,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912379 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:19,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:19,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911923 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:21,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:21,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091208 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:22:21,427::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:22:21,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000624479 s, 594 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:23,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:23,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913569 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:22:23,553::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:22:23,564::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068014 s, 769 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:22:23,733::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:22:23,734::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:22:25,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:25,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09122 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:27,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:27,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091589 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:22:28,277::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:22:29,572::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:22:29,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:29,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911997 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:22:31,442::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:22:31,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661905 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:31,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:32,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543699 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:22:33,568::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:22:33,580::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716089 s, 730 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:22:33,742::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:22:33,743::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:22:34,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:34,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546372 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:37,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:37,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541166 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:39,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:39,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091405 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:22:41,457::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:22:41,469::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692191 s, 536 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:41,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:41,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911664 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:22:43,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:22:43,596::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000714729 s, 732 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:22:43,755::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:22:43,756::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:22:43,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:43,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912426 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:22:44,584::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:22:45,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:46,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108197 s, 9.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:48,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:48,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915594 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:50,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:50,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559796 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:22:51,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:22:51,484::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649071 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:52,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:52,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914475 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:22:53,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:22:53,611::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000644858 s, 811 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:22:53,762::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:22:53,764::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:22:55,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:55,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914518 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:22:57,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:57,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912048 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:22:58,320::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:22:59,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:22:59,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091682 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:22:59,595::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:23:01,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:01,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912715 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:23:01,487::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:23:01,499::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069411 s, 534 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:03,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:03,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911242 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:23:03,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:23:03,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000622821 s, 840 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:23:03,775::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:23:03,776::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:23:05,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:05,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913629 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:07,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:07,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912446 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:09,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:09,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912544 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:23:11,503::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:23:11,514::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000710128 s, 522 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:11,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:12,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550046 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:23:13,629::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:23:13,645::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:23:13,647::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:23:13,650::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:23:13,651::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:23:13,651::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:23:13,652::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:23:13,653::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:23:13,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00061508 s, 850 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:23:13,779::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:23:13,780::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:23:14,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:23:14,598::task::595::Storage.TaskManager.Task::(_updateState) Task=`5866d534-a36f-40ea-9a6c-ff2c39fe7299`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:23:14,599::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:23:14,620::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:23:14,621::task::1191::Storage.TaskManager.Task::(prepare) Task=`5866d534-a36f-40ea-9a6c-ff2c39fe7299`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:23:14,621::task::595::Storage.TaskManager.Task::(_updateState) Task=`5866d534-a36f-40ea-9a6c-ff2c39fe7299`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:23:14,622::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:23:14,622::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:23:14,623::task::993::Storage.TaskManager.Task::(_decref) Task=`5866d534-a36f-40ea-9a6c-ff2c39fe7299`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:23:14,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0960753 s, 10.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:23:14,654::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:23:16,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:17,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559508 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:19,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:19,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911386 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:21,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:21,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912966 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:23:21,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:23:21,531::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673415 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:23,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:23,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912217 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:23:23,668::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:23:23,679::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670898 s, 780 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:23:23,791::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:23:23,792::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:23:25,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:26,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558041 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:28,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:23:28,353::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:23:29,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36508 s, 750 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:23:29,667::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:23:31,534::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:23:31,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063684 s, 583 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:31,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:31,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912832 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:33,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:23:33,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:33,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914235 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:23:33,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0109359 s, 47.8 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:23:33,806::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:23:33,810::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:23:35,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:35,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912874 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:37,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:38,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913234 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:40,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:40,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914153 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:23:41,548::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:23:41,576::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:23:41,578::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:23:41,581::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:23:41,584::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:23:41,584::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:23:41,584::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:23:41,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:23:41,597::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067263 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:42,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:42,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091225 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:23:43,796::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:23:43,807::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659908 s, 793 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:23:43,817::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:23:43,818::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:23:44,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:44,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913466 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:23:44,679::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:23:46,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:46,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970512 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:48,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:49,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551441 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:51,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:51,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0930585 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:23:51,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:23:51,612::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000701297 s, 529 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:53,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:53,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913582 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:23:53,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:23:53,823::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696717 s, 751 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:23:53,829::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:23:53,830::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:23:55,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:55,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911794 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:23:57,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:23:57,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919188 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:23:58,386::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:23:59,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:23:59,691::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:24:00,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.51856 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:24:01,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:24:01,627::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659012 s, 563 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:02,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:02,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913779 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:24:03,827::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:24:03,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696635 s, 751 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:24:03,843::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:24:03,849::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:24:04,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:04,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557864 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:06,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:06,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914411 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:08,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:09,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912268 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:11,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:11,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912199 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:24:11,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:24:11,641::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000603336 s, 615 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:13,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:13,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911834 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:24:13,855::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:24:13,871::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000775488 s, 674 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:24:13,870::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:24:13,877::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:24:14,694::task::595::Storage.TaskManager.Task::(_updateState) Task=`c3b01dc9-3ea7-4dfb-8c0c-83165233fa45`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:24:14,695::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:24:14,698::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:24:14,698::task::1191::Storage.TaskManager.Task::(prepare) Task=`c3b01dc9-3ea7-4dfb-8c0c-83165233fa45`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:24:14,698::task::595::Storage.TaskManager.Task::(_updateState) Task=`c3b01dc9-3ea7-4dfb-8c0c-83165233fa45`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:24:14,698::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:24:14,698::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:24:14,698::task::993::Storage.TaskManager.Task::(_decref) Task=`c3b01dc9-3ea7-4dfb-8c0c-83165233fa45`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:24:14,726::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:24:15,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:15,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913361 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:17,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:18,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545911 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:20,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:20,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913946 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:24:21,645::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:24:21,657::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648419 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:22,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:22,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0908895 s, 11.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:24:23,866::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:24:23,867::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:24:23,884::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:24:23,895::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000623074 s, 839 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:24,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:24,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913618 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:26,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:26,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911376 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:24:28,419::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:24:28,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:29,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519933 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:24:29,739::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:24:31,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:31,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542974 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:24:31,660::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:24:31,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651728 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:33,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:33,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912671 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:24:33,879::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:24:33,880::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:24:33,899::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:24:33,911::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000718251 s, 728 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:35,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:36,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549686 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:38,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:38,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912637 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:40,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:40,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091301 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:24:41,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:24:41,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679604 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:42,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:42,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912238 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:24:43,891::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:24:43,892::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:24:43,915::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:24:43,927::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656295 s, 797 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:44,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:24:44,751::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:24:44,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091174 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:46,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:46,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912976 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:48,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:48,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912401 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:51,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:51,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919445 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:24:51,691::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:24:51,703::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653844 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:53,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:53,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912257 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:24:53,897::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:24:53,898::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:24:54,253::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:24:54,264::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646359 s, 809 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:55,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:55,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913124 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:24:57,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:24:57,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912724 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:24:58,458::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:24:59,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:24:59,764::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:25:00,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556495 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:25:01,706::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:25:01,718::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687595 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:02,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:02,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913753 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:25:03,909::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:25:03,910::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:25:04,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:25:04,283::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:25:04,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692359 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:04,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108056 s, 9.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:06,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:06,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912669 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:08,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:08,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912157 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:10,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:11,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.498804 s, 2.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:25:11,721::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:25:11,733::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000703942 s, 527 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:13,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:25:13,918::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:25:13,919::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:25:14,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558056 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:25:14,309::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:25:14,321::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665538 s, 786 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:25:14,768::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d388e8e-2059-425c-a4b3-487eefa4c3f3`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:25:14,768::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:25:14,848::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:25:14,848::task::1191::Storage.TaskManager.Task::(prepare) Task=`2d388e8e-2059-425c-a4b3-487eefa4c3f3`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:25:14,848::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d388e8e-2059-425c-a4b3-487eefa4c3f3`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:25:14,849::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:25:14,849::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:25:14,849::task::993::Storage.TaskManager.Task::(_decref) Task=`2d388e8e-2059-425c-a4b3-487eefa4c3f3`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:25:14,877::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:25:16,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:16,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.140565 s, 7.3 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:18,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:18,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:20,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:20,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912313 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:25:21,737::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:25:21,749::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654678 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:22,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:22,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912455 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:25:23,929::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:25:23,930::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:25:24,325::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:25:24,336::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655099 s, 798 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:24,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:24,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911631 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:26,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:27,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510529 s, 2.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:25:28,489::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:25:29,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:29,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938861 s, 10.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:25:29,890::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:25:31,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:31,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912748 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:25:31,752::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:25:31,764::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664684 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:33,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:25:33,941::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:25:33,942::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:25:34,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.376052 s, 2.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:25:34,340::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:25:34,353::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687418 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:36,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:36,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913042 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:38,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:38,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543249 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:40,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:41,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556026 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:25:41,767::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:25:41,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670127 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:43,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:43,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091208 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:25:43,952::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:25:43,953::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:25:44,357::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:25:44,369::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000654981 s, 798 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:25:44,902::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:25:45,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:45,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912847 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:47,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:47,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912087 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:49,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:49,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911756 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:25:51,782::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:25:51,794::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069241 s, 536 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:51,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:51,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919315 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:53,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:25:53,965::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:25:53,966::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:25:54,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:25:54,499::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00150455 s, 348 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:54,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.563374 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:25:56,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:56,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913892 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:25:58,517::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:25:58,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:25:58,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912361 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:25:59,915::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:26:00,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:01,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542469 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:26:01,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:26:01,808::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657872 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:03,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:03,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912308 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:26:03,977::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:26:03,978::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:26:04,508::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:26:04,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675499 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:05,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:05,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913358 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:07,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:07,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913624 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:09,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:09,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914263 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:26:11,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:26:11,824::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000673926 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:11,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:12,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912163 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:26:13,989::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:26:13,991::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:26:14,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:14,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913467 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:26:14,526::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:26:14,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650314 s, 804 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:26:14,918::task::595::Storage.TaskManager.Task::(_updateState) Task=`7cbe23d2-14ef-4680-b35b-851457e6bee4`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:26:14,918::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:26:14,921::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:26:14,922::task::1191::Storage.TaskManager.Task::(prepare) Task=`7cbe23d2-14ef-4680-b35b-851457e6bee4`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:26:14,922::task::595::Storage.TaskManager.Task::(_updateState) Task=`7cbe23d2-14ef-4680-b35b-851457e6bee4`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:26:14,922::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:26:14,922::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:26:14,922::task::993::Storage.TaskManager.Task::(_decref) Task=`7cbe23d2-14ef-4680-b35b-851457e6bee4`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:26:14,948::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:26:16,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:16,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.396429 s, 2.6 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:18,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:18,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913709 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:20,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:20,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912368 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:26:21,827::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:26:21,839::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653772 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:22,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:22,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912984 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:26:24,002::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:26:24,003::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:26:24,543::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:26:24,555::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000619949 s, 844 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:24,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:25,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091298 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:27,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:27,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911743 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:26:28,557::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:26:29,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:29,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914565 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:26:29,960::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:26:31,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:31,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912708 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:26:31,842::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:26:31,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653264 s, 568 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:33,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:33,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911591 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:26:34,014::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:26:34,015::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:26:34,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:26:34,578::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000732251 s, 714 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:35,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:35,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913061 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:37,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:37,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912254 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:39,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:39,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913727 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:26:41,857::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:41,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:26:41,879::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000725352 s, 511 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:42,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.539676 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:26:44,020::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:26:44,021::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:26:44,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:44,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915577 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:26:44,582::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:26:44,594::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674624 s, 775 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:26:44,972::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:26:46,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:46,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911592 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:48,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:48,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912759 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:50,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:50,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911797 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:26:51,883::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:26:51,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067549 s, 549 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:52,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:53,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.512351 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:26:54,027::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:26:54,028::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:26:54,598::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:26:54,610::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000740606 s, 706 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:55,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:55,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918913 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:26:57,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:57,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915678 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:26:58,590::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:26:59,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:26:59,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913452 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:26:59,984::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:27:01,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:01,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912927 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:27:01,899::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:27:01,912::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643631 s, 576 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:03,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:04,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918726 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:27:04,039::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:27:04,042::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:27:04,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:27:04,625::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713625 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:06,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:06,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913375 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:08,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:08,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912892 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:10,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:10,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912511 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:27:11,917::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:27:11,928::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654049 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:12,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:12,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911904 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:27:14,048::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:27:14,049::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:27:14,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:14,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914375 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:27:14,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:27:14,644::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000758251 s, 690 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:27:14,987::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f2b993d-601b-4035-9d64-ded8af376bf5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:27:14,988::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:27:14,992::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:27:14,992::task::1191::Storage.TaskManager.Task::(prepare) Task=`8f2b993d-601b-4035-9d64-ded8af376bf5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:27:14,992::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f2b993d-601b-4035-9d64-ded8af376bf5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:27:14,992::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:27:14,992::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:27:14,993::task::993::Storage.TaskManager.Task::(_decref) Task=`8f2b993d-601b-4035-9d64-ded8af376bf5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:27:15,018::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:27:16,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:16,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915582 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:18,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:18,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913282 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:20,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:21,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912171 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:27:21,932::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:27:21,944::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667928 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:23,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:23,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560505 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:27:24,059::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:27:24,060::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:27:24,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:27:24,666::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675529 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:25,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:25,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091333 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:27,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:27,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912576 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:27:28,621::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:27:29,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:30,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912044 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:27:30,035::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:27:31,947::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:27:31,959::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623236 s, 595 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:32,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:32,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911789 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:27:34,071::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:27:34,072::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:27:34,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:34,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091225 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:27:34,670::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:27:34,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000738606 s, 708 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:36,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:36,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559418 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:38,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:39,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.133484 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:41,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:41,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55127 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:27:41,962::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:27:41,974::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680506 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:43,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:43,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912539 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:27:44,084::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:27:44,085::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:27:44,687::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:27:44,699::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065284 s, 801 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:27:45,047::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:27:45,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:47,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36182 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:49,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:49,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911593 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:51,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:51,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912572 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:27:51,978::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:27:51,989::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000615414 s, 603 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:53,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:53,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916362 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:27:54,096::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:27:54,097::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:27:54,704::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:27:54,715::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651215 s, 803 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:55,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:55,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091304 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:27:57,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:57,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913113 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:27:58,654::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:27:59,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:27:59,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912568 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:28:00,059::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:28:01,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:01,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102093 s, 10.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:28:01,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:28:02,010::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000723322 s, 513 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:04,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:28:04,103::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:28:04,104::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:28:04,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519994 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:28:04,719::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:28:04,731::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695875 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:06,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:06,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091413 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:08,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:08,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914048 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:10,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:10,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912594 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:28:12,018::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:28:12,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000719806 s, 515 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:12,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:13,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911687 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:28:14,122::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:28:14,123::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:28:14,733::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:28:14,749::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:28:14,751::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:28:14,754::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:28:14,755::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:28:14,755::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:28:14,756::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:28:14,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:28:14,768::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000615886 s, 849 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:28:15,062::task::595::Storage.TaskManager.Task::(_updateState) Task=`c16cd50e-e57b-4ceb-80de-e05c306bfdc0`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:28:15,062::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:28:15,065::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:28:15,065::task::1191::Storage.TaskManager.Task::(prepare) Task=`c16cd50e-e57b-4ceb-80de-e05c306bfdc0`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:28:15,065::task::595::Storage.TaskManager.Task::(_updateState) Task=`c16cd50e-e57b-4ceb-80de-e05c306bfdc0`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:28:15,066::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:28:15,066::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:28:15,066::task::993::Storage.TaskManager.Task::(_decref) Task=`c16cd50e-e57b-4ceb-80de-e05c306bfdc0`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:28:15,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:28:15,103::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:28:15,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913437 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:17,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:17,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913238 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:19,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:19,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912176 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:21,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:21,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912491 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:28:22,033::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:28:22,046::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630906 s, 588 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:23,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:24,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54168 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:28:24,133::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:28:24,134::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:28:24,773::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:28:24,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636571 s, 822 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:26,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:26,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558788 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:28:28,687::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:28:28,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:28,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091193 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:28:30,115::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:28:30,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:30,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911936 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:28:32,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:28:32,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688473 s, 539 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:32,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:33,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912612 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:28:34,145::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:28:34,147::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:28:34,789::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:28:34,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000726222 s, 720 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:35,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:35,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912579 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:37,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:37,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912427 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:39,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:39,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0941862 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:41,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:41,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0941142 s, 10.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:28:42,063::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:28:42,076::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:28:42,077::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:28:42,080::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:28:42,081::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:28:42,082::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:28:42,082::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:28:42,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:28:42,094::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647234 s, 573 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:43,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:43,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912956 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:28:44,155::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:28:44,156::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:28:44,804::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:28:44,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068641 s, 762 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:28:45,127::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:28:45,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:46,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.567771 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:48,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:48,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912671 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:50,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:50,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914022 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:28:52,098::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:28:52,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642338 s, 578 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:52,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:53,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558257 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:28:54,166::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:28:54,167::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:28:54,822::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:28:54,834::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629646 s, 831 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:55,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:55,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558192 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:28:57,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:28:58,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091196 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:28:58,719::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:29:00,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:29:00,140::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:29:00,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545876 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:29:02,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:29:02,124::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000695382 s, 534 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:02,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:02,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914739 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:29:04,178::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:29:04,179::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:29:04,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:29:04,879::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:04,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103617 s, 9.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:29:04,902::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00572224 s, 91.4 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:06,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:07,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913398 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:09,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:09,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915954 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:11,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:11,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914841 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:29:12,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:29:12,141::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672841 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:13,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:13,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913192 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:29:14,191::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:29:14,192::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:29:14,909::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:29:14,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000711328 s, 735 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:29:15,143::task::595::Storage.TaskManager.Task::(_updateState) Task=`d6224b61-b2ec-40eb-8df0-cd264e021297`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:29:15,143::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:29:15,146::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:29:15,147::task::1191::Storage.TaskManager.Task::(prepare) Task=`d6224b61-b2ec-40eb-8df0-cd264e021297`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:29:15,147::task::595::Storage.TaskManager.Task::(_updateState) Task=`d6224b61-b2ec-40eb-8df0-cd264e021297`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:29:15,147::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:29:15,147::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:29:15,147::task::993::Storage.TaskManager.Task::(_decref) Task=`d6224b61-b2ec-40eb-8df0-cd264e021297`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:29:15,175::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:29:15,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:15,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913255 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:17,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:17,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913321 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:19,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:19,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916551 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:21,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:22,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911329 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:29:22,145::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:29:22,157::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654278 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:24,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:24,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911785 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:29:24,202::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:29:24,203::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:29:24,925::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:29:24,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641384 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:26,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:26,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912906 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:28,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:28,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913285 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:29:28,759::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-113::DEBUG::2014-07-14 17:29:30,187::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:29:30,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:30,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091187 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:29:32,161::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:29:32,172::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00061068 s, 608 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:32,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:33,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555954 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:29:34,213::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:29:34,214::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:29:34,942::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:29:34,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000723959 s, 722 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:35,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:35,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54608 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:37,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:37,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912324 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:39,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:40,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558396 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:29:42,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:29:42,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662196 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:42,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:42,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544074 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:29:44,221::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:29:44,223::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:29:44,957::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:44,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:29:44,981::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666916 s, 784 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:45,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921815 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:29:45,199::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:29:47,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:47,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132625 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:49,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:49,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911553 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:51,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:51,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912805 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:29:52,191::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:29:52,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658524 s, 563 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:53,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:53,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913243 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:29:54,234::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:29:54,235::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:29:55,087::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:29:55,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000722919 s, 723 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:55,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:55,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911973 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:29:57,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:57,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913117 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:29:58,792::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:29:59,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:29:59,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919114 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:30:00,213::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:30:01,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:02,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913271 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:30:02,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:30:02,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665646 s, 557 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:04,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:30:04,242::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:30:04,243::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:30:04,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546225 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:30:05,121::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:30:05,132::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641508 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:06,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:06,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913054 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:08,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:08,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912398 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:10,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:11,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912881 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:30:12,221::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:30:12,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665917 s, 557 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:13,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:13,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914322 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:30:14,254::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:30:14,255::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:30:15,136::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:30:15,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000644091 s, 812 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:15,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:30:15,216::task::595::Storage.TaskManager.Task::(_updateState) Task=`41eb99ee-9270-4191-8b0a-1d50ff35b467`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:30:15,216::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:30:15,749::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:30:15,751::task::1191::Storage.TaskManager.Task::(prepare) Task=`41eb99ee-9270-4191-8b0a-1d50ff35b467`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:30:15,753::task::595::Storage.TaskManager.Task::(_updateState) Task=`41eb99ee-9270-4191-8b0a-1d50ff35b467`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:30:15,754::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:30:15,755::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:30:15,757::task::993::Storage.TaskManager.Task::(_decref) Task=`41eb99ee-9270-4191-8b0a-1d50ff35b467`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:30:15,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.56865 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:30:15,797::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:30:17,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:17,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912787 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:19,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:20,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911726 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:22,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:22,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911109 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:30:22,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:30:22,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066269 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:24,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:30:24,262::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:30:24,263::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:30:24,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911558 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:30:25,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:30:25,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000654834 s, 799 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:26,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:26,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091323 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:28,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:28,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912393 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:30:28,825::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:30:30,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:30,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914016 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:30:30,810::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:30:32,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:30:32,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663038 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:32,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:32,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919061 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:30:34,274::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:30:34,275::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:30:34,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:34,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914035 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:30:35,169::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:30:35,181::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671249 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:36,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:37,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544337 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:39,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:40,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547971 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:42,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:42,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912485 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:30:42,266::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:30:42,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674271 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:44,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:30:44,283::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:30:44,284::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:30:44,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.461823 s, 2.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:30:45,185::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:30:45,197::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00101906 s, 513 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:30:45,825::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:30:46,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:46,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912382 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:48,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:48,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913241 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:50,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:51,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915395 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:30:52,281::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:30:52,293::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705572 s, 526 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:53,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:53,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912647 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:30:54,292::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:30:54,293::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:30:55,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:30:55,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:30:55,268::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706288 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:55,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.113396 s, 9.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:30:57,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:57,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912799 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:30:58,857::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:30:59,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:30:59,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912207 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:31:00,838::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:31:01,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:01,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910978 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:31:02,296::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:31:02,308::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000719319 s, 516 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:03,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:04,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542623 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:31:04,305::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:31:04,307::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:31:05,347::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:31:05,358::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670852 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:06,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:06,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913474 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:08,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:08,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913561 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:10,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:10,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912738 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:31:12,311::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:31:12,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678275 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:12,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:12,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091149 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:31:14,317::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:31:14,319::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:31:14,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:14,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923079 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:31:15,363::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:31:15,374::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000647859 s, 807 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:31:15,842::task::595::Storage.TaskManager.Task::(_updateState) Task=`8cb7bfd8-cb91-4b02-beb0-1e2e6805c3c3`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:31:15,843::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:31:15,848::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:31:15,850::task::1191::Storage.TaskManager.Task::(prepare) Task=`8cb7bfd8-cb91-4b02-beb0-1e2e6805c3c3`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:31:15,858::task::595::Storage.TaskManager.Task::(_updateState) Task=`8cb7bfd8-cb91-4b02-beb0-1e2e6805c3c3`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:31:15,859::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:31:15,860::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:31:15,861::task::993::Storage.TaskManager.Task::(_decref) Task=`8cb7bfd8-cb91-4b02-beb0-1e2e6805c3c3`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:31:15,900::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:31:16,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:16,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912651 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:18,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:19,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558712 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:21,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:21,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912186 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:31:22,327::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:31:22,339::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626094 s, 593 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:23,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:23,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912751 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:31:24,328::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:31:24,329::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:31:25,379::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:31:25,390::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681315 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:25,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:25,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912795 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:27,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:27,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913392 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:31:28,888::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:31:29,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:30,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912898 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:31:30,914::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:31:32,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:32,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912759 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:31:32,342::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:31:32,353::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000631478 s, 588 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:34,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:34,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091237 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:31:34,336::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:31:34,337::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:31:35,394::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:31:35,406::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000716715 s, 730 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:36,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:36,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912612 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:38,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:39,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557034 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:41,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:41,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913174 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:31:42,357::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:31:42,368::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670741 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:43,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:43,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543448 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:31:44,348::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:31:44,349::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:31:45,413::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:31:45,428::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629556 s, 831 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:45,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:45,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912042 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:31:45,929::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:31:47,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:48,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914435 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:50,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:50,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557654 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:31:52,372::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:31:52,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672443 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:52,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:52,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912274 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:31:54,360::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:31:54,361::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:31:54,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:54,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091055 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:31:55,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:31:55,446::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707469 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:31:56,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:56,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913245 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:31:58,921::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:31:58,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:31:59,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913219 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:32:00,950::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:32:01,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:01,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913438 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:32:02,388::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:32:02,399::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654151 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:03,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:03,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913281 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:32:04,372::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:32:04,374::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:32:05,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:05,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912654 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:32:05,449::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:32:05,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000739665 s, 707 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:07,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:07,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0987016 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:09,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:09,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911799 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:11,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:11,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091321 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:32:12,403::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:32:12,414::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630388 s, 589 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:13,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:13,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912618 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:32:14,385::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:32:14,386::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:32:15,466::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:32:15,478::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662226 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:15,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:32:15,954::task::595::Storage.TaskManager.Task::(_updateState) Task=`f235eede-5d8b-42fb-b017-b2beee7bceb4`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:32:15,955::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:32:16,029::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:32:16,030::task::1191::Storage.TaskManager.Task::(prepare) Task=`f235eede-5d8b-42fb-b017-b2beee7bceb4`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:32:16,031::task::595::Storage.TaskManager.Task::(_updateState) Task=`f235eede-5d8b-42fb-b017-b2beee7bceb4`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:32:16,032::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:32:16,033::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:32:16,034::task::993::Storage.TaskManager.Task::(_decref) Task=`f235eede-5d8b-42fb-b017-b2beee7bceb4`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:32:16,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10109 s, 10.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:32:16,071::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:32:18,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:18,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917607 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:20,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:20,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912025 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:22,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:32:22,487::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:22,501::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103405 s, 9.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:32:22,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00162092 s, 229 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:32:24,397::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:32:24,398::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:32:24,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:24,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912333 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:32:25,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:32:25,495::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656533 s, 797 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:26,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:26,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912647 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:28,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:32:28,954::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:32:29,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919734 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:32:31,084::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:32:31,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:31,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0934063 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:32:32,509::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:32:32,521::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00080034 s, 464 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:33,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:33,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912576 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:32:34,409::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:32:34,411::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:32:35,498::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:32:35,510::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000744862 s, 702 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:35,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:35,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913132 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:37,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:37,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913809 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:39,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:40,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544148 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:42,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:32:42,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:32:42,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000727202 s, 510 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:42,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.107838 s, 9.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:32:44,418::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:32:44,419::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:32:44,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:44,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912173 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:32:45,514::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:32:45,526::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066044 s, 792 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:32:46,098::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:32:46,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:46,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913911 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:48,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:48,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.112569 s, 9.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:51,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:51,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913041 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:32:52,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:32:52,633::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699045 s, 531 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:53,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:53,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0998703 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:32:54,428::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:32:54,429::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:32:55,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:55,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913118 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:32:55,530::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:32:55,541::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657394 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:32:57,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:57,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132377 s, 7.7 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:32:58,988::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:32:59,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:32:59,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091078 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:33:01,110::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:33:01,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:01,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913012 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:33:02,636::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:33:02,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00069134 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:03,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:04,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55669 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:33:04,440::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:33:04,441::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:33:05,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:33:05,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659697 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:06,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:06,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09125 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:08,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:08,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911213 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:10,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:10,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912766 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:33:12,651::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:33:12,662::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000612365 s, 606 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:12,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:12,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910584 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:33:14,447::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:33:14,449::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:33:14,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:14,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924375 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:33:15,559::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:33:15,611::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:33:15,613::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:33:15,617::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:33:15,621::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:33:15,622::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:33:15,622::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:33:15,625::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:33:15,638::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673712 s, 776 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:33:16,114::task::595::Storage.TaskManager.Task::(_updateState) Task=`8382bcd2-37d5-4629-b2bd-aaaeff926254`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:33:16,115::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:33:16,118::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:33:16,119::task::1191::Storage.TaskManager.Task::(prepare) Task=`8382bcd2-37d5-4629-b2bd-aaaeff926254`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:33:16,120::task::595::Storage.TaskManager.Task::(_updateState) Task=`8382bcd2-37d5-4629-b2bd-aaaeff926254`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:33:16,121::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:33:16,121::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:33:16,122::task::993::Storage.TaskManager.Task::(_decref) Task=`8382bcd2-37d5-4629-b2bd-aaaeff926254`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:33:16,154::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:33:16,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:17,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914576 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:19,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:19,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912089 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:21,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:21,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913136 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:33:22,666::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:33:22,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000624503 s, 594 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:23,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:23,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091228 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:33:24,452::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:33:24,453::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:33:25,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:25,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912569 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:33:25,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:33:25,658::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662419 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:27,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:27,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912207 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:33:29,020::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:33:29,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:30,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.532586 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:33:31,167::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:33:32,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:32,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915514 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:33:32,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:33:32,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650527 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:34,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:33:34,462::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:33:34,464::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:33:35,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55806 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:33:35,662::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:33:35,674::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706122 s, 741 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:37,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:37,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918504 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:39,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:39,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5451 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:41,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:42,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555014 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:33:42,696::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:33:42,711::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:33:42,712::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:33:42,715::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:33:42,717::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:33:42,717::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:33:42,717::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:33:42,718::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:33:42,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620232 s, 598 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:44,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:44,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091419 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:33:44,475::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:33:44,476::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 17:33:45,679::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:33:45,693::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646428 s, 809 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:33:46,182::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:33:46,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:46,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0975223 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:48,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:49,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544393 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:51,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:51,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916512 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:33:52,733::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:33:52,745::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652588 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:53,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:53,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922961 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:33:54,487::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:33:54,488::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:33:55,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:33:56,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:33:56,093::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00071084 s, 736 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:56,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.527853 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:33:58,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:33:58,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542532 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:33:59,052::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:34:00,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:00,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913787 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:34:01,194::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:34:02,748::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:34:02,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686877 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:02,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:02,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912755 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:34:04,499::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:34:04,500::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:34:04,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:05,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912167 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:34:06,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:34:06,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000736979 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:07,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:07,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913269 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:09,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:09,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911922 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:11,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:11,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913097 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:34:12,764::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:34:12,776::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682711 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:13,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:13,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091323 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:34:14,512::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:34:14,513::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:34:15,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:16,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544794 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:34:16,123::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:34:16,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626834 s, 834 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:34:16,197::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa331a86-292f-4403-b45f-538cb07831bc`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:34:16,198::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:34:16,203::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:34:16,204::task::1191::Storage.TaskManager.Task::(prepare) Task=`fa331a86-292f-4403-b45f-538cb07831bc`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:34:16,205::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa331a86-292f-4403-b45f-538cb07831bc`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:34:16,206::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:34:16,207::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:34:16,209::task::993::Storage.TaskManager.Task::(_decref) Task=`fa331a86-292f-4403-b45f-538cb07831bc`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:34:16,240::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:34:18,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:18,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543756 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:20,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:20,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925304 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:34:22,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:34:22,792::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000619723 s, 599 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:22,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:23,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910696 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:34:24,524::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:34:24,525::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:34:25,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:25,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914771 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:34:26,149::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:34:26,161::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000628825 s, 832 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:27,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:27,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911832 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:34:29,085::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:34:29,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:29,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911586 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:34:31,254::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:34:31,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:31,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910938 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:34:32,795::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:34:32,807::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646672 s, 574 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:33,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:33,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913206 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:34:34,536::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:34:34,537::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:34:35,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:34:37,016::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:37,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.41154 s, 725 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:34:37,034::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00592746 s, 88.2 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:39,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:39,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54129 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:41,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:42,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549267 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:34:42,810::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:34:42,827::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677712 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:44,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:44,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912626 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:34:44,548::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:34:44,549::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-113::DEBUG::2014-07-14 17:34:46,268::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:34:46,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:46,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100409 s, 10.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:34:47,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:34:47,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672034 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:48,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:49,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559743 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:51,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:51,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912169 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:34:52,835::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:34:52,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645976 s, 574 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:53,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:53,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911625 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:34:54,557::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:34:54,558::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:34:55,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:55,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.254905 s, 4.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:34:57,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:34:57,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691128 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:34:57,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:57,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913727 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:34:59,118::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:34:59,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:34:59,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911257 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:35:01,281::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:35:01,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:02,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910189 s, 11.3 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:35:02,850::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:35:02,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000615588 s, 603 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:04,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:04,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914857 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:35:04,568::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:35:04,570::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:35:06,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:06,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925742 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:35:07,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:35:07,087::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673821 s, 776 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:08,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:08,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552541 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:10,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:10,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911969 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:35:12,865::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:35:12,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635228 s, 584 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:12,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:13,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910845 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:35:14,578::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:35:14,579::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:35:15,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:15,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913509 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:35:16,285::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6d9f3bf-c2ae-41f8-a1b5-0416a5cd3a4c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:35:16,286::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:35:16,333::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:35:16,334::task::1191::Storage.TaskManager.Task::(prepare) Task=`f6d9f3bf-c2ae-41f8-a1b5-0416a5cd3a4c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:35:16,335::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6d9f3bf-c2ae-41f8-a1b5-0416a5cd3a4c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:35:16,335::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:35:16,336::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:35:16,337::task::993::Storage.TaskManager.Task::(_decref) Task=`f6d9f3bf-c2ae-41f8-a1b5-0416a5cd3a4c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:35:16,372::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:35:17,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:35:17,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000736232 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:17,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:17,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091137 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:19,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:19,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091224 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:21,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:21,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911617 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:35:22,880::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:35:22,892::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627125 s, 592 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:23,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:23,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913177 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:35:24,590::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:35:24,591::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:35:25,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:25,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912207 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:35:27,106::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:35:27,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668494 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:27,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:27,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:35:29,154::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:35:29,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:29,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912238 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:35:31,385::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:35:32,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:32,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911992 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:35:32,895::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:35:32,907::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650909 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:34,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:35:34,599::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:35:34,600::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:35:34,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.512443 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:36,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:36,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913988 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:35:37,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:35:37,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679838 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:38,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:38,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913024 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:40,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:40,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914154 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:35:42,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:35:42,921::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667442 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:43,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:43,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0930084 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:35:44,609::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:35:44,610::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:35:45,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:45,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912401 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:35:46,405::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:35:47,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:35:47,149::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000618102 s, 846 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:47,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:47,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54308 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:49,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:49,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912236 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:51,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:52,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912137 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:35:52,925::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:35:52,937::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664984 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:54,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:54,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557612 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:35:54,629::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:35:54,632::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:35:56,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:35:56,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915867 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:35:57,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:35:57,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006829 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:35:58,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:35:59,186::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:35:59,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558003 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:01,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:36:01,419::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:36:01,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912677 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:36:02,940::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:36:02,952::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00061317 s, 605 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:03,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:03,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918513 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:36:04,639::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:36:04,640::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:36:05,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:05,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911723 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:36:07,171::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:36:07,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683265 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:07,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:07,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912676 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:09,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:09,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106037 s, 9.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:11,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:12,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914653 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:36:12,955::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:36:12,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000634563 s, 585 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:14,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:14,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09129 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:36:14,643::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:36:14,644::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:36:16,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:36:16,423::task::595::Storage.TaskManager.Task::(_updateState) Task=`675af984-72dd-4372-8e6c-ed52f9a54407`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:36:16,423::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:36:16,761::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:36:16,761::task::1191::Storage.TaskManager.Task::(prepare) Task=`675af984-72dd-4372-8e6c-ed52f9a54407`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:36:16,761::task::595::Storage.TaskManager.Task::(_updateState) Task=`675af984-72dd-4372-8e6c-ed52f9a54407`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:36:16,762::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:36:16,762::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:36:16,762::task::993::Storage.TaskManager.Task::(_decref) Task=`675af984-72dd-4372-8e6c-ed52f9a54407`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:36:16,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554476 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:36:16,791::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:36:17,186::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:36:17,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671586 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:18,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:18,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917249 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:20,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:21,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911641 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:36:22,970::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:36:22,982::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000638756 s, 581 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:23,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:23,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913535 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:36:24,653::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:36:24,654::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:36:25,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:25,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557789 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:36:27,202::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:36:27,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671529 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:27,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:27,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918349 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:36:29,223::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:36:29,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:29,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091221 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:36:31,806::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:36:31,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:32,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910665 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:36:32,985::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:36:32,997::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626208 s, 592 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:34,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:34,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.110809 s, 9.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:36:34,663::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:36:34,664::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:36:36,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:36,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912993 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:36:37,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:36:37,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655944 s, 797 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:38,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:38,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912767 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:40,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:40,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918706 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:42,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:42,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912807 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:36:43,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:36:43,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685601 s, 541 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:36:44,672::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:36:44,673::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:36:44,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:45,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.488383 s, 2.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:36:46,820::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:36:47,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:36:47,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664774 s, 787 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:47,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:47,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912621 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:49,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:49,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911332 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:51,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:51,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911423 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:36:53,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:36:53,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066696 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:53,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:54,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556003 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:36:54,684::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:36:54,685::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:36:56,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:56,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912553 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:36:57,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:36:57,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000705836 s, 741 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:36:58,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:36:58,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911796 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:36:59,263::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:37:00,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:00,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910958 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:37:01,839::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:37:02,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:37:03,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:37:03,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00130284 s, 285 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:03,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.105095 s, 9.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:37:04,696::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:37:04,697::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:37:05,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:05,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543019 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:37:07,266::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:37:07,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690076 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:07,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:07,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911465 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:09,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:09,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914268 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:11,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:11,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912886 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:37:13,057::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:37:13,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000609721 s, 608 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:13,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:14,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912169 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:37:14,703::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:37:14,705::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:37:16,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:16,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913115 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:37:16,842::task::595::Storage.TaskManager.Task::(_updateState) Task=`0e2111de-aab1-4337-a8dc-0e410249a5e5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:37:16,842::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:37:16,846::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:37:16,846::task::1191::Storage.TaskManager.Task::(prepare) Task=`0e2111de-aab1-4337-a8dc-0e410249a5e5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:37:16,846::task::595::Storage.TaskManager.Task::(_updateState) Task=`0e2111de-aab1-4337-a8dc-0e410249a5e5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:37:16,846::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:37:16,846::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:37:16,846::task::993::Storage.TaskManager.Task::(_decref) Task=`0e2111de-aab1-4337-a8dc-0e410249a5e5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:37:16,874::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:37:17,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:37:17,294::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657273 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:18,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:18,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912679 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:20,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:20,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911576 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:22,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:22,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913912 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:37:23,072::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:37:23,085::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639905 s, 580 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:37:24,713::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:37:24,714::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:37:24,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:24,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912226 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:26,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:27,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912834 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:37:27,298::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:37:27,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670831 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:29,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:29,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915931 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:37:29,311::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:37:31,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:31,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091317 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:37:31,886::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:37:33,088::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:37:33,101::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641854 s, 578 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:33,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:33,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911689 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:37:34,725::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:37:34,726::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:37:35,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:35,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912618 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:37:37,314::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:37:37,325::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688431 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:37,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:37,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911984 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:39,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:39,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912929 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:41,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:41,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912176 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:37:43,104::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:37:43,116::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686501 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:43,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:44,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091165 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:37:44,733::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:37:44,734::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:37:46,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:46,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557821 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:37:46,898::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:37:47,329::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:37:47,342::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068072 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:48,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:49,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550799 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:51,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:51,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912633 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:37:53,120::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:37:53,132::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652678 s, 568 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:53,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:53,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923468 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:37:54,745::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:37:54,746::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:37:55,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:55,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922168 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:37:57,346::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:37:57,358::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703812 s, 743 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:37:57,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:57,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912495 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:37:59,358::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:37:59,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:37:59,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911915 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:01,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:01,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922859 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:38:01,910::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:38:03,135::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:38:03,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656308 s, 565 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:03,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:04,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912907 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:38:04,757::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:38:04,758::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:38:06,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:06,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913057 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:38:07,362::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:38:07,373::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068715 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:08,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:08,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919568 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:10,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:10,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913307 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:12,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:12,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920748 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:38:13,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:38:13,161::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678546 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:14,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:14,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912037 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:38:14,771::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:38:14,772::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:38:16,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:16,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912696 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:38:16,913::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a7d83f6-cde9-4c18-b337-fa3197980ddd`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:38:16,913::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:38:16,916::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:38:16,916::task::1191::Storage.TaskManager.Task::(prepare) Task=`4a7d83f6-cde9-4c18-b337-fa3197980ddd`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:38:16,916::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a7d83f6-cde9-4c18-b337-fa3197980ddd`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:38:16,916::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:38:16,917::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:38:16,917::task::993::Storage.TaskManager.Task::(_decref) Task=`4a7d83f6-cde9-4c18-b337-fa3197980ddd`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:38:16,942::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:38:17,375::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:38:17,389::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:38:17,390::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:38:17,393::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:38:17,394::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:38:17,395::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:38:17,395::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:38:17,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:38:17,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670328 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:18,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:18,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912999 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:20,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:20,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926332 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:22,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:23,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911813 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:38:23,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:38:23,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643104 s, 577 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:38:24,779::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:38:24,780::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:38:25,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:25,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912655 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:27,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:27,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912793 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:38:27,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:38:27,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667959 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:29,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:38:29,395::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:38:29,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912299 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:31,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:31,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912459 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:38:31,954::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:38:33,181::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:38:33,192::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655805 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:33,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:33,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911255 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:38:34,796::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:38:34,801::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:38:35,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:35,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091342 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:38:37,427::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:38:37,438::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660943 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:37,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:37,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091329 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:39,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:39,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913968 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:42,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:42,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091218 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:38:43,194::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:38:43,207::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:38:43,208::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:38:43,211::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:38:43,213::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:38:43,213::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:38:43,213::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:38:43,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:38:43,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643066 s, 577 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:44,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:44,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913983 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:38:44,807::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:38:44,808::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:38:46,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:46,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919605 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:38:46,966::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:38:47,442::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:38:47,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064321 s, 813 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:48,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:48,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915467 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:50,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:50,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911658 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:52,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:38:53,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:53,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.527151 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:38:53,339::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000755265 s, 491 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:38:54,819::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:38:54,820::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:38:55,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:55,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914121 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:38:57,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:38:57,470::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000719817 s, 727 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:38:57,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:57,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910827 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:38:59,428::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:38:59,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:38:59,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912303 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:01,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:01,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091222 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:39:01,979::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:39:03,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:39:03,356::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686711 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:03,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:03,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912645 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:39:04,829::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:39:04,831::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:39:05,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:06,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504914 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:39:07,474::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:39:07,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690708 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:08,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:09,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544743 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:11,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:11,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913634 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:13,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:13,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912994 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:39:13,360::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:39:13,371::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646638 s, 574 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:39:14,840::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:39:14,842::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:39:15,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:15,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0945345 s, 10.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:39:16,982::task::595::Storage.TaskManager.Task::(_updateState) Task=`deb2e82d-0c80-4cbb-b89f-b8ed071601a5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:39:16,982::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:39:16,985::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:39:16,985::task::1191::Storage.TaskManager.Task::(prepare) Task=`deb2e82d-0c80-4cbb-b89f-b8ed071601a5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:39:16,985::task::595::Storage.TaskManager.Task::(_updateState) Task=`deb2e82d-0c80-4cbb-b89f-b8ed071601a5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:39:16,986::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:39:16,986::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:39:16,986::task::993::Storage.TaskManager.Task::(_decref) Task=`deb2e82d-0c80-4cbb-b89f-b8ed071601a5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:39:17,013::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:39:17,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:39:17,972::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:39:17,985::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000625982 s, 835 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:17,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.571765 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:20,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:20,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912363 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:22,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:22,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542431 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:39:23,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:39:23,725::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621294 s, 597 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:24,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:39:24,853::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:39:24,854::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:39:25,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.536381 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:27,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:27,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922816 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:39:27,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:39:28,006::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000671379 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:29,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:39:29,462::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:39:29,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918008 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:31,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:31,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912667 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:39:32,026::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:39:33,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:33,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912735 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:39:33,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:39:33,739::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658055 s, 564 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:39:34,864::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:39:34,865::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:39:35,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:35,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09135 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:37,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:37,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913154 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:39:38,011::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:39:38,022::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066007 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:39,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:40,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913402 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:42,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:42,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912713 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:39:43,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:39:43,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660731 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:44,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:44,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916556 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:39:44,874::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:39:44,875::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:39:46,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:46,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0971068 s, 10.5 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:39:47,038::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:39:48,027::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:39:48,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648488 s, 806 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:48,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:49,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551628 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:51,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:51,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918132 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:53,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:53,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912334 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:39:53,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:39:53,769::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00064077 s, 579 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:39:54,886::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:39:54,887::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:39:55,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:55,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911848 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:39:57,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:57,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911869 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:39:58,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:39:58,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706379 s, 740 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:39:59,495::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:39:59,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:39:59,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912802 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:01,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:01,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911867 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:40:02,056::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:40:03,773::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:40:03,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000696207 s, 533 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:03,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:03,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912948 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:40:04,898::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:40:04,900::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:40:05,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:06,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913832 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:40:08,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:08,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:40:08,081::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000763228 s, 685 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:08,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916818 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:10,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:10,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911565 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:12,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:12,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913566 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:40:13,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:40:13,799::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066244 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:14,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:40:14,910::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:40:14,911::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:40:14,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544809 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:16,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:40:17,059::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c75abd5-a590-4673-9f99-4778e0c7d999`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:40:17,060::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:40:17,095::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:40:17,096::task::1191::Storage.TaskManager.Task::(prepare) Task=`1c75abd5-a590-4673-9f99-4778e0c7d999`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:40:17,097::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c75abd5-a590-4673-9f99-4778e0c7d999`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:40:17,097::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:40:17,098::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:40:17,099::task::993::Storage.TaskManager.Task::(_decref) Task=`1c75abd5-a590-4673-9f99-4778e0c7d999`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:40:17,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0957104 s, 10.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:40:17,128::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:40:18,085::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:40:18,097::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000715412 s, 731 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:19,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:19,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914129 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:21,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:21,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912791 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:23,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:23,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912454 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:40:23,803::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:40:23,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000625796 s, 593 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:40:24,922::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:40:24,923::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:40:25,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:25,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913265 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:27,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:27,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912811 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:40:28,101::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:40:28,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690001 s, 758 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:40:29,528::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:40:29,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:29,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922515 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:31,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:31,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912335 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:40:32,140::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:40:33,819::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:40:33,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000619607 s, 599 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:33,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:34,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918276 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:40:34,930::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:40:34,932::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:40:36,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:36,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097332 s, 10.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:40:38,116::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:40:38,128::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693217 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:38,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:38,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912529 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:40,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:40,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913892 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:42,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:42,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912104 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:40:43,834::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:40:43,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690855 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:44,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:40:44,940::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:40:44,941::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:40:45,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557786 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:47,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:40:47,153::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:40:48,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913593 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:40:48,132::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:40:48,144::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666278 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:50,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:50,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912258 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:52,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:52,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911642 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:40:53,849::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:40:53,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689935 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:54,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:54,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911447 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:40:54,950::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:40:54,951::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:40:56,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:56,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913199 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:40:58,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:40:58,160::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680251 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:40:58,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:40:58,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912143 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:40:59,562::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:41:00,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:00,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912837 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:41:02,165::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:41:02,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:02,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911912 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:41:03,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:41:03,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676807 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:04,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:04,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911559 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:41:04,967::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:41:04,971::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:41:06,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:07,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091344 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:41:08,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:41:08,175::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661406 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:09,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:09,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911844 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:11,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:11,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091351 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:13,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:13,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912271 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:41:13,879::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:41:13,891::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000699469 s, 530 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:41:14,976::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:41:14,977::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:41:15,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:15,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100334 s, 10.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:41:17,168::task::595::Storage.TaskManager.Task::(_updateState) Task=`58fb0032-d374-45f3-8d6d-25bba222e36c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:41:17,169::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:41:17,172::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:41:17,172::task::1191::Storage.TaskManager.Task::(prepare) Task=`58fb0032-d374-45f3-8d6d-25bba222e36c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:41:17,172::task::595::Storage.TaskManager.Task::(_updateState) Task=`58fb0032-d374-45f3-8d6d-25bba222e36c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:41:17,172::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:41:17,172::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:41:17,173::task::993::Storage.TaskManager.Task::(_decref) Task=`58fb0032-d374-45f3-8d6d-25bba222e36c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:41:17,199::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:41:17,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:17,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914226 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:41:18,178::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:41:18,190::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692181 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:19,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:19,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912029 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:21,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:22,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911974 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:41:23,895::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:41:23,907::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618226 s, 600 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:24,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:24,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550607 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:41:24,986::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:41:24,987::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:41:26,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:26,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912741 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:41:28,195::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:41:28,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660034 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:28,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:28,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912368 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:41:29,593::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:41:30,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:30,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913057 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:41:32,211::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:41:32,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:33,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912423 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:41:33,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:41:33,923::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637776 s, 582 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:41:34,993::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:41:35,000::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:41:35,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:35,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911742 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:37,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:37,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913095 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:41:38,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:41:38,221::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693213 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:39,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:39,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557462 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:41,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:42,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912572 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:41:43,926::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:41:43,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654148 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:44,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:44,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914673 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:41:45,005::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:41:45,006::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:41:46,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:46,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0962006 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:41:47,224::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:41:48,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:41:48,237::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653369 s, 800 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:48,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:48,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912216 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:50,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:51,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913105 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:53,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:53,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913099 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:41:53,942::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:41:53,953::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688524 s, 539 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:41:55,015::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:41:55,016::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:41:55,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:55,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911433 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:41:57,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:57,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912961 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:41:58,241::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:41:58,252::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679986 s, 769 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:41:59,627::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:41:59,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:41:59,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913884 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:01,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:01,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0978566 s, 10.5 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:42:02,235::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:42:03,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:03,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:42:03,982::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000785758 s, 472 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:04,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091458 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:42:05,023::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:42:05,024::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:42:06,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:06,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913498 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:08,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:42:08,770::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:42:08,783::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000638313 s, 819 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:08,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.572792 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:10,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:10,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913966 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:12,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:13,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912016 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:42:13,986::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:42:13,997::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00064641 s, 574 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:42:15,035::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:42:15,036::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:42:15,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:15,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.570465 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:42:17,238::task::595::Storage.TaskManager.Task::(_updateState) Task=`772a276a-cc1e-48e5-bf1d-0da233630cbb`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:42:17,239::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:42:17,241::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:42:17,242::task::1191::Storage.TaskManager.Task::(prepare) Task=`772a276a-cc1e-48e5-bf1d-0da233630cbb`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:42:17,242::task::595::Storage.TaskManager.Task::(_updateState) Task=`772a276a-cc1e-48e5-bf1d-0da233630cbb`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:42:17,242::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:42:17,242::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:42:17,242::task::993::Storage.TaskManager.Task::(_decref) Task=`772a276a-cc1e-48e5-bf1d-0da233630cbb`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:42:17,269::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:42:17,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:17,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913148 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:42:18,791::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:42:18,802::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660642 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:19,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:19,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912537 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:22,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:22,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914209 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:42:24,001::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:42:24,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00062081 s, 598 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:24,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:24,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912444 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:42:25,044::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:42:25,046::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:42:26,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:26,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09136 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:28,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:28,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913284 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:42:28,807::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:42:28,818::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000684137 s, 764 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:42:29,659::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:42:30,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:30,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913564 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:42:32,281::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:42:32,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:33,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913186 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:42:34,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:42:34,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000622705 s, 596 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:42:35,057::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:42:35,058::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:42:35,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:35,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914163 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:37,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:37,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912327 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:42:38,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:42:38,833::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070337 s, 744 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:39,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:39,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542595 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:41,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:42,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091592 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:42:44,034::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:42:44,045::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000632695 s, 586 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:44,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:44,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913503 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:42:45,064::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:42:45,066::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:42:46,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:46,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914647 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:42:47,293::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:42:48,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:42:49,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:42:49,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656665 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:49,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.575916 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:51,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:51,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558182 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:53,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:53,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914734 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:42:54,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:42:54,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000695993 s, 533 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:42:55,077::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:42:55,078::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:42:55,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:55,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912268 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:42:57,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:42:58,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927901 s, 11.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:42:59,087::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:42:59,098::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000701972 s, 745 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:42:59,700::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:43:00,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:00,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911294 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:02,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:43:02,306::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:43:02,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912501 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:43:04,064::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:43:04,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000714907 s, 519 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:04,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:04,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912497 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:43:05,089::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:43:05,090::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:43:06,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:06,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912496 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:08,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:08,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912374 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:43:09,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:43:09,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693057 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:10,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:10,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912672 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:12,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:12,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912473 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:43:14,079::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:43:14,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672212 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:14,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:15,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912723 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:43:15,101::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:43:15,103::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:43:17,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:17,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091386 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:43:17,309::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c6d001c-9c65-4280-958b-6cf386c45e98`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:43:17,309::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:43:17,313::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:43:17,313::task::1191::Storage.TaskManager.Task::(prepare) Task=`4c6d001c-9c65-4280-958b-6cf386c45e98`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:43:17,313::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c6d001c-9c65-4280-958b-6cf386c45e98`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:43:17,313::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:43:17,313::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:43:17,313::task::993::Storage.TaskManager.Task::(_decref) Task=`4c6d001c-9c65-4280-958b-6cf386c45e98`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:43:17,341::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 17:43:19,116::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:43:19,130::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:43:19,131::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:43:19,134::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:43:19,136::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:43:19,136::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:43:19,136::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:43:19,137::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:43:19,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661941 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:19,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:19,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542948 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:21,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:21,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914428 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:23,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:23,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913808 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:43:24,094::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:43:24,106::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617143 s, 601 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:43:25,111::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:43:25,112::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:43:25,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:26,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912563 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:28,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:28,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545125 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:43:29,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:43:29,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066179 s, 790 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:43:29,733::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:43:30,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:30,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912702 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:43:32,353::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:43:32,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:32,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09142 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:43:34,109::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:43:34,121::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662594 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:34,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:34,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091191 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:43:35,124::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:43:35,125::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:43:37,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:37,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923997 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:39,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:43:39,232::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:39,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10361 s, 9.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:43:39,250::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00565584 s, 92.5 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:41,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:41,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912075 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:43,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:43,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912769 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:43:44,123::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:43:44,136::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:43:44,137::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:43:44,140::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:43:44,142::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:43:44,142::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:43:44,142::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:43:44,143::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:43:44,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661667 s, 561 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:43:45,136::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:43:45,137::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:43:45,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:45,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914503 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:43:47,366::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:43:47,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:47,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912334 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:43:49,262::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:43:49,273::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632803 s, 826 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:49,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:49,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911538 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:51,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:52,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912512 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:54,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:43:54,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:54,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103118 s, 9.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:43:54,193::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00443531 s, 83.6 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:43:55,146::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:43:55,147::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:43:56,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:56,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911554 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:43:58,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:43:58,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091286 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:43:59,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:43:59,289::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070276 s, 744 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:43:59,765::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:44:00,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:01,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550868 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:44:02,378::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:44:03,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:03,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912085 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:44:04,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:44:04,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688952 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:05,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:44:05,159::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:44:05,160::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:44:05,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911243 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:07,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:08,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36272 s, 751 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:44:09,293::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:44:09,305::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699715 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:10,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:10,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911764 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:12,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:13,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556838 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:44:14,213::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:44:14,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693698 s, 535 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:44:15,167::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:44:15,168::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:44:15,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:15,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913079 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:44:17,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`cc40540d-e580-48b2-bcd3-350d5db95275`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:44:17,382::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:44:17,385::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:44:17,385::task::1191::Storage.TaskManager.Task::(prepare) Task=`cc40540d-e580-48b2-bcd3-350d5db95275`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:44:17,385::task::595::Storage.TaskManager.Task::(_updateState) Task=`cc40540d-e580-48b2-bcd3-350d5db95275`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:44:17,385::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:44:17,386::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:44:17,386::task::993::Storage.TaskManager.Task::(_decref) Task=`cc40540d-e580-48b2-bcd3-350d5db95275`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:44:17,415::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:44:17,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:18,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541059 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:44:19,309::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:44:19,321::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681045 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:20,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:20,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911688 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:22,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:22,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912737 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:44:24,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:44:24,241::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068353 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:24,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:24,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550458 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:44:25,178::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:44:25,180::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:44:26,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:26,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912901 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:28,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:29,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912231 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:44:29,325::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:44:29,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000633678 s, 825 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:44:29,798::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:44:31,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:31,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913101 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:44:32,428::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:44:33,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:33,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913756 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:44:34,244::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:44:34,256::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649694 s, 571 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:44:35,191::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:44:35,192::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:44:35,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:35,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557506 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:37,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:37,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912221 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:44:39,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:44:39,352::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670849 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:40,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:40,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915315 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:42,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:42,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913816 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:44,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:44:44,348::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:44:44,361::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00168202 s, 221 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:44,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103825 s, 9.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:44:45,199::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:44:45,200::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:44:46,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:46,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55695 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:44:47,444::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:44:49,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:49,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091262 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:44:49,356::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:44:49,368::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665372 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:51,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:51,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912962 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:53,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:53,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914826 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:44:54,367::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:44:54,379::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646322 s, 574 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:44:55,211::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:44:55,212::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:44:55,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:55,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912367 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:57,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:44:57,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913288 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:44:59,372::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:44:59,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000750098 s, 697 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:44:59,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:44:59,829::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:45:00,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559586 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:02,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:02,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0957896 s, 10.7 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:45:02,456::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:45:04,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:45:04,443::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:04,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104883 s, 9.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:45:04,464::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00790164 s, 47.0 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:45:05,224::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:45:05,225::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:45:06,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:06,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913245 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:08,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:08,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921209 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:45:09,388::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:45:09,400::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000708762 s, 738 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:10,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:10,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912794 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:12,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:12,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091202 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:45:14,470::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:45:14,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067706 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:14,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:45:15,237::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:45:15,238::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:45:15,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554036 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:45:17,458::task::595::Storage.TaskManager.Task::(_updateState) Task=`f9c9fc17-52b3-4986-8dde-f40fe7d5b61b`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:45:17,459::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:45:17,462::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:45:17,462::task::1191::Storage.TaskManager.Task::(prepare) Task=`f9c9fc17-52b3-4986-8dde-f40fe7d5b61b`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:45:17,462::task::595::Storage.TaskManager.Task::(_updateState) Task=`f9c9fc17-52b3-4986-8dde-f40fe7d5b61b`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:45:17,462::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:45:17,463::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:45:17,463::task::993::Storage.TaskManager.Task::(_decref) Task=`f9c9fc17-52b3-4986-8dde-f40fe7d5b61b`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:45:17,493::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:45:17,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:18,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.534455 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:45:19,404::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:45:19,415::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696837 s, 751 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:20,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:20,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546269 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:22,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:22,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915921 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:45:24,486::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:45:24,498::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000623601 s, 595 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:24,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:24,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912167 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:45:25,248::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:45:25,249::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:45:26,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:27,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913234 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:29,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:29,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913611 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:45:29,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:45:29,431::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657736 s, 795 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:45:29,865::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:45:31,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:31,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913107 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:45:32,505::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:45:33,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:33,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910619 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:45:34,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:45:34,514::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674503 s, 550 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:45:35,261::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:45:35,262::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:45:35,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:35,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091227 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:37,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:37,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911905 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:45:39,500::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:45:39,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692525 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:39,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:39,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914163 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:41,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:41,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912736 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:43,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:43,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091431 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:45:44,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:45:44,528::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661345 s, 561 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:45:45,274::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:45:45,275::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:45:45,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:46,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100946 s, 10.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:45:47,521::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:45:48,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:48,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091298 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:45:49,516::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:45:49,528::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000604271 s, 866 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:50,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:50,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914186 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:52,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:52,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913321 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:45:54,532::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:45:54,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680587 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:54,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:54,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910842 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:45:55,285::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:45:55,286::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:45:56,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:56,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912053 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:45:58,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:45:58,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915498 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:45:59,779::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:45:59,790::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666503 s, 785 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:45:59,898::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:46:00,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:01,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091177 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:46:02,538::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:46:03,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:03,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914634 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:46:04,547::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:46:04,559::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677758 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:05,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:05,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912485 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:46:05,295::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:46:05,296::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:46:07,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:07,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913631 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:09,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:09,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912456 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:46:09,794::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:46:09,806::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000670945 s, 779 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:11,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:11,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912526 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:13,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:14,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542934 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:46:14,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:46:14,574::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065801 s, 564 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:46:15,307::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:46:15,309::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:46:16,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:46:17,540::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f597676-3f90-4c7e-bcc0-3b1a65aef7be`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:46:17,541::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:46:17,562::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:46:17,563::task::1191::Storage.TaskManager.Task::(prepare) Task=`5f597676-3f90-4c7e-bcc0-3b1a65aef7be`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:46:17,564::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f597676-3f90-4c7e-bcc0-3b1a65aef7be`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:46:17,565::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:46:17,565::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:46:17,567::task::993::Storage.TaskManager.Task::(_decref) Task=`5f597676-3f90-4c7e-bcc0-3b1a65aef7be`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:46:17,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37094 s, 747 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:46:17,598::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:46:19,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:46:20,145::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:46:20,158::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068676 s, 762 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:20,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.570593 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:22,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:22,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911884 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:24,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:46:24,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:24,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.568968 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:46:24,881::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661288 s, 561 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:46:25,318::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:46:25,319::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:46:26,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:27,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544639 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:29,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:29,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912481 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:46:29,930::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:46:30,167::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:46:30,178::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000724233 s, 722 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:31,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:32,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559145 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:46:32,610::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:46:34,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:34,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912952 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:46:34,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:46:34,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000658837 s, 563 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:46:35,331::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:46:35,332::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:46:36,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:36,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913109 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:38,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:38,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54425 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:46:40,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:46:40,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677201 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:40,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:41,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912491 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:43,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:43,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912886 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:46:44,902::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:46:44,913::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000608536 s, 610 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:46:45,338::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:46:45,339::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:46:45,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:45,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100029 s, 10.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:46:47,623::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:46:47,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:47,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911718 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:49,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:50,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912959 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:46:50,198::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:46:50,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000675868 s, 774 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:52,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:52,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915038 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:54,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:54,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091229 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:46:54,918::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:46:54,929::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682265 s, 544 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:46:55,350::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:46:55,352::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:46:56,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:57,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37122 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:46:59,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:46:59,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913019 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:46:59,964::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:47:00,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:47:00,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000702338 s, 745 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:01,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:01,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916259 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:47:02,635::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:47:03,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:04,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911061 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:47:04,932::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:47:04,944::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000638467 s, 581 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:47:05,362::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:47:05,364::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:47:06,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:06,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920856 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:08,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:08,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912126 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:47:10,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:47:10,242::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678509 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:10,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:10,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912972 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:12,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:12,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091246 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:14,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:14,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913753 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:47:14,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:47:14,959::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663047 s, 560 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:47:15,374::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:47:15,375::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:47:16,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:16,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912188 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:47:17,638::task::595::Storage.TaskManager.Task::(_updateState) Task=`30ace084-0c96-455f-a68e-5ece3a726923`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:47:17,638::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:47:17,642::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:47:17,642::task::1191::Storage.TaskManager.Task::(prepare) Task=`30ace084-0c96-455f-a68e-5ece3a726923`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:47:17,642::task::595::Storage.TaskManager.Task::(_updateState) Task=`30ace084-0c96-455f-a68e-5ece3a726923`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:47:17,642::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:47:17,642::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:47:17,642::task::993::Storage.TaskManager.Task::(_decref) Task=`30ace084-0c96-455f-a68e-5ece3a726923`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:47:17,669::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:47:18,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:18,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912369 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:47:20,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:47:20,257::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695941 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:20,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:21,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912017 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:23,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:23,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912817 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:47:24,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:47:24,977::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000930465 s, 399 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:25,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:25,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912516 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:47:25,384::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:47:25,385::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:47:27,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:27,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.512136 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:29,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:47:29,996::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:47:30,428::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:47:30,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000750278 s, 697 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:30,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.573796 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:32,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:47:32,682::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:47:33,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36455 s, 750 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:47:34,982::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:47:34,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000702922 s, 528 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:47:35,396::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:47:35,398::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:47:35,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:36,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546435 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:38,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:38,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913367 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:47:40,446::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:47:40,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677911 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:40,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:40,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091269 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:42,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:43,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550035 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:47:44,997::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:47:45,008::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662492 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:45,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:45,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914553 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:47:45,402::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:47:45,404::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:47:47,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:47,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091471 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:47:47,695::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:47:49,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:49,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913731 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:47:50,462::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:47:50,474::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000663095 s, 789 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:51,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:52,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548658 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:54,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:54,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912666 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:47:55,012::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:47:55,024::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066014 s, 562 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:47:55,408::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:47:55,410::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:47:56,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:57,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39748 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:47:59,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:47:59,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915193 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:48:00,028::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:48:00,478::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:48:00,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069015 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:01,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:02,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558863 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:48:02,707::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:48:04,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:04,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546507 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:48:05,421::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:48:05,422::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:48:05,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:48:05,534::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629654 s, 589 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:06,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:07,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914078 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:09,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:09,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091208 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:48:10,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:48:10,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707683 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:11,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:11,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913378 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:13,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:13,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912735 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:48:15,428::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:48:15,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:48:15,444::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:48:15,549::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:15,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104548 s, 9.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:48:15,563::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000939234 s, 395 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:48:17,711::task::595::Storage.TaskManager.Task::(_updateState) Task=`01c4e271-43ff-47ed-abb7-03a1d0b260fb`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:48:17,711::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:48:17,714::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:48:17,714::task::1191::Storage.TaskManager.Task::(prepare) Task=`01c4e271-43ff-47ed-abb7-03a1d0b260fb`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:48:17,714::task::595::Storage.TaskManager.Task::(_updateState) Task=`01c4e271-43ff-47ed-abb7-03a1d0b260fb`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:48:17,714::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:48:17,714::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:48:17,715::task::993::Storage.TaskManager.Task::(_decref) Task=`01c4e271-43ff-47ed-abb7-03a1d0b260fb`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:48:17,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:48:17,753::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:48:17,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913523 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:19,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:20,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557288 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:48:20,508::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:48:20,543::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:48:20,545::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:48:20,549::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:48:20,552::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:48:20,553::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:48:20,553::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:48:20,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:48:20,570::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062244 s, 840 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:22,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:22,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913813 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:24,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:24,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916454 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:48:25,439::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:48:25,441::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:48:25,570::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:48:25,582::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636142 s, 583 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:26,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:26,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911529 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:28,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:29,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911846 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:48:30,059::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:48:30,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:48:30,606::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000702925 s, 744 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:31,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:31,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091371 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:48:32,765::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:48:33,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:33,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912937 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:35,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:35,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132438 s, 7.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:48:35,452::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:48:35,453::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:48:35,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:48:35,598::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000633787 s, 585 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:37,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:37,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913722 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:39,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:39,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091389 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:48:40,610::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:48:40,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713947 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:41,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:41,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927363 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:43,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:43,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913331 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:48:45,460::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:48:45,461::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:48:45,600::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:48:45,631::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:48:45,633::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:48:45,638::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:48:45,643::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:48:45,643::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:48:45,644::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:48:45,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:48:45,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704417 s, 527 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:45,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:46,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.096307 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:48:47,777::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:48:48,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:48,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912945 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:50,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:50,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091274 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:48:50,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:48:50,638::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642894 s, 814 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:52,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:52,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912591 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:54,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:54,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091214 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:48:55,472::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:48:55,473::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:48:55,667::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:48:55,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000687219 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:56,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:56,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912332 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:48:58,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:48:58,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913084 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:49:00,090::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:49:00,642::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:49:00,653::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000659871 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:00,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:00,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922605 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:49:02,789::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:49:03,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:03,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54194 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:49:05,479::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:49:05,480::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:49:05,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:49:06,133::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:49:06,145::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000710248 s, 522 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:06,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.564859 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:08,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:08,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914448 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:10,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:49:10,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:49:10,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656563 s, 797 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:10,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.537925 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:12,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:12,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912214 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:14,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:15,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912738 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:49:15,489::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:49:15,490::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:49:16,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:49:16,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00485394 s, 76.4 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:17,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:17,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912768 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:49:17,792::task::595::Storage.TaskManager.Task::(_updateState) Task=`b845daf0-7bb4-4313-b6bc-40c1b4016076`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:49:17,793::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:49:17,796::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:49:17,796::task::1191::Storage.TaskManager.Task::(prepare) Task=`b845daf0-7bb4-4313-b6bc-40c1b4016076`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:49:17,796::task::595::Storage.TaskManager.Task::(_updateState) Task=`b845daf0-7bb4-4313-b6bc-40c1b4016076`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:49:17,796::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:49:17,796::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:49:17,796::task::993::Storage.TaskManager.Task::(_decref) Task=`b845daf0-7bb4-4313-b6bc-40c1b4016076`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:49:17,821::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:49:19,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:19,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913008 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:49:20,834::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:49:20,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00150973 s, 346 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:21,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:21,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560325 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:23,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:24,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919827 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:49:25,501::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:49:25,502::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:49:26,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:26,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923952 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:49:26,175::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:49:26,188::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621958 s, 597 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:28,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:28,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913462 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:49:30,124::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:49:30,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:30,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912918 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:49:30,850::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:49:30,862::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661649 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:32,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:32,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912226 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:49:32,832::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:49:34,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:34,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912812 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:49:35,514::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:49:35,515::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:49:36,191::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:49:36,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006332 s, 586 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:36,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:36,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914356 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:38,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:39,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.509307 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:49:40,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:49:40,877::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707688 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:41,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:41,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915494 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:43,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:43,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916647 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:45,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:49:45,521::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:49:45,523::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:49:45,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912663 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:49:46,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:49:46,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669785 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:47,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:47,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091316 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:49:47,846::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:49:49,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:49,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913222 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:49:50,882::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:49:50,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000639673 s, 818 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:51,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:52,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911762 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:54,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:54,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913431 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:49:55,533::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:49:55,534::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:49:56,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:49:56,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:49:56,312::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628115 s, 591 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:56,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108254 s, 9.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:49:58,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:49:58,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911324 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:50:00,157::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:50:00,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:50:01,024::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:01,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.568055 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:50:01,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00656858 s, 79.6 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:50:02,865::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:50:03,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:03,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915377 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:05,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:05,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912228 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:50:05,545::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:50:05,546::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-77::DEBUG::2014-07-14 17:50:06,319::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:50:06,330::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664434 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:07,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:07,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914805 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:09,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:09,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921649 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:50:11,052::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:50:11,064::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067019 s, 780 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:11,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:12,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555952 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:14,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:14,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912568 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:50:15,557::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:50:15,558::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:50:16,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:50:17,619::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:50:17,632::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000740044 s, 501 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:17,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39583 s, 734 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:50:17,868::task::595::Storage.TaskManager.Task::(_updateState) Task=`d1e48c35-457f-4070-b789-174ca7b45b59`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:50:17,868::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:50:17,871::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:50:17,871::task::1191::Storage.TaskManager.Task::(prepare) Task=`d1e48c35-457f-4070-b789-174ca7b45b59`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:50:17,871::task::595::Storage.TaskManager.Task::(_updateState) Task=`d1e48c35-457f-4070-b789-174ca7b45b59`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:50:17,872::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:50:17,872::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:50:17,872::task::993::Storage.TaskManager.Task::(_decref) Task=`d1e48c35-457f-4070-b789-174ca7b45b59`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:50:17,899::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:50:19,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:20,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551929 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:50:21,068::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:50:21,079::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000641216 s, 816 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:22,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:22,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938768 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:24,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:24,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912338 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:50:25,567::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:50:25,568::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:50:26,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:50:27,852::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:27,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38767 s, 738 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:50:27,869::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00532414 s, 69.7 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:29,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:29,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912582 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:50:30,190::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:50:31,084::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:50:31,096::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000638502 s, 819 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:31,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:32,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913801 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:50:32,912::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:50:34,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:34,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550456 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:50:35,579::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:50:35,580::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:50:36,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:36,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0936136 s, 10.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:50:37,874::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:50:37,887::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000636166 s, 583 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:38,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:38,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913749 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:40,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:41,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914297 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:50:41,099::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:50:41,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668134 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:43,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:43,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912872 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:45,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:45,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916508 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:50:45,595::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:50:45,599::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:50:47,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:47,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912475 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:50:47,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:50:47,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655805 s, 566 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:50:47,924::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:50:49,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:49,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911405 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:50:51,115::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:50:51,127::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707009 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:51,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:53,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549781 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:55,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:55,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912894 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:50:55,605::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:50:55,607::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:50:57,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:57,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913233 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:50:57,905::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:50:57,917::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670785 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:50:59,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:50:59,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913178 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:51:00,224::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:51:01,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:51:01,143::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069863 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:01,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:01,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914304 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:51:02,937::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:51:03,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:03,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912937 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:51:05,613::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:51:05,614::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:51:05,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:05,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912641 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:07,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:07,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913855 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:51:07,922::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:51:07,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654925 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:09,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:10,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915434 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:51:11,148::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:51:11,159::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677908 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:12,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:12,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915579 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:14,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:14,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913126 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:51:15,624::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:51:15,628::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:51:16,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:16,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913947 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:51:17,942::task::595::Storage.TaskManager.Task::(_updateState) Task=`88ee63be-e750-4ea9-9bb1-ea3366bcd388`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:51:17,943::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-77::DEBUG::2014-07-14 17:51:17,945::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-113::INFO::2014-07-14 17:51:17,956::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:51:17,956::task::1191::Storage.TaskManager.Task::(prepare) Task=`88ee63be-e750-4ea9-9bb1-ea3366bcd388`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:51:17,957::task::595::Storage.TaskManager.Task::(_updateState) Task=`88ee63be-e750-4ea9-9bb1-ea3366bcd388`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:51:17,957::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:51:17,957::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:51:17,957::task::993::Storage.TaskManager.Task::(_decref) Task=`88ee63be-e750-4ea9-9bb1-ea3366bcd388`::ref 0 aborting False >Thread-77::DEBUG::2014-07-14 17:51:17,958::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705066 s, 526 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:51:17,986::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:51:18,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:18,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912714 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:20,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:20,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913086 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:51:21,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:51:21,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065373 s, 800 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:22,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:22,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912543 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:24,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:25,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545798 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:51:25,635::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:51:25,636::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:51:27,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:27,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910906 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:51:27,963::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:51:27,975::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652254 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:29,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:30,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544588 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:51:30,256::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:51:31,179::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:51:31,190::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666167 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:32,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:32,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912501 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:51:32,998::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:51:34,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:34,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911903 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:51:35,643::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:51:35,644::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:51:36,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:36,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913056 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:51:37,978::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:51:37,990::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000620742 s, 598 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:38,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:38,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917772 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:40,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:40,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913349 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:51:41,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:51:41,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695395 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:42,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:42,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914403 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:44,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:44,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913739 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:51:45,658::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:51:45,662::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:51:46,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:47,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912788 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:51:47,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:51:48,005::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000639888 s, 580 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:51:48,014::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:51:49,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:49,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911747 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:51:51,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:51,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:51:51,236::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695341 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:51,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.17949 s, 5.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:53,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:53,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912744 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:55,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:55,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911644 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:51:55,669::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:51:55,670::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:51:57,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:57,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913261 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:51:58,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:51:58,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649768 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:51:59,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:51:59,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091234 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:52:00,293::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:52:01,329::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:52:01,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698199 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:01,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:02,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912246 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:52:03,029::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:52:04,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:04,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912969 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:52:05,680::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:52:05,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:52:06,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:06,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913534 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:52:08,025::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:52:08,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000711769 s, 521 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:08,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:08,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912085 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:10,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:10,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091434 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:52:11,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:52:11,356::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000705179 s, 742 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:12,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:12,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913417 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:14,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:14,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0940322 s, 10.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:52:15,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:52:15,698::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:52:16,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:16,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913922 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:52:18,038::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c377ffd-94eb-4c24-ae1c-41f8a9c47b7d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:52:18,038::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-77::DEBUG::2014-07-14 17:52:18,041::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-113::INFO::2014-07-14 17:52:18,052::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:52:18,052::task::1191::Storage.TaskManager.Task::(prepare) Task=`2c377ffd-94eb-4c24-ae1c-41f8a9c47b7d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:52:18,053::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c377ffd-94eb-4c24-ae1c-41f8a9c47b7d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:52:18,053::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:52:18,053::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:52:18,053::task::993::Storage.TaskManager.Task::(_decref) Task=`2c377ffd-94eb-4c24-ae1c-41f8a9c47b7d`::ref 0 aborting False >Thread-77::DEBUG::2014-07-14 17:52:18,055::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653741 s, 568 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:52:18,079::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:52:18,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:18,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103197 s, 9.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:20,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:21,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913295 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:52:21,360::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:52:21,372::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679847 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:23,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:23,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915361 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:25,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:25,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091353 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:52:25,705::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:52:25,706::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:52:27,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:27,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.115617 s, 8.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:52:28,059::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:52:28,071::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645053 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:29,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:29,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913719 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:52:30,325::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:52:31,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:52:31,388::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674997 s, 775 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:31,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:31,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916107 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:52:33,090::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:52:33,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:33,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912175 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:52:35,717::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:52:35,719::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:52:35,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:36,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914188 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:52:38,075::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:52:38,087::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621516 s, 597 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:38,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:38,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912565 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:40,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:40,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913678 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:52:41,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:52:41,403::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683695 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:42,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:42,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914486 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:44,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:44,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913173 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:52:45,729::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:52:45,733::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:52:46,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:47,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518518 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:52:48,090::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:52:48,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689195 s, 538 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:52:48,111::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:52:49,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:49,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912813 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:52:51,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:52:51,419::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000654133 s, 800 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:51,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:53,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39821 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:55,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:55,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913446 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:52:55,740::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:52:55,741::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:52:57,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:57,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911996 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:52:58,106::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:52:58,124::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00597817 s, 62.1 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:52:59,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:52:59,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913068 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:53:00,359::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:53:01,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:53:01,435::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678347 s, 771 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:01,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:01,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912726 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:53:03,127::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:53:03,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:03,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912941 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:53:05,747::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:53:05,748::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:53:05,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:05,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913095 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:07,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:08,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912091 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:53:08,131::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:53:08,143::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670774 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:10,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:10,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913291 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:53:11,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:53:11,450::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672555 s, 778 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:12,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:12,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913054 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:14,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:14,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912856 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:53:15,761::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:53:15,764::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:53:16,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:16,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999696 s, 10.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:53:18,131::task::595::Storage.TaskManager.Task::(_updateState) Task=`08c72ec7-d88e-4db5-ba64-199b48d69eea`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:53:18,131::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:53:18,134::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:53:18,134::task::1191::Storage.TaskManager.Task::(prepare) Task=`08c72ec7-d88e-4db5-ba64-199b48d69eea`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:53:18,134::task::595::Storage.TaskManager.Task::(_updateState) Task=`08c72ec7-d88e-4db5-ba64-199b48d69eea`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:53:18,134::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:53:18,134::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:53:18,135::task::993::Storage.TaskManager.Task::(_decref) Task=`08c72ec7-d88e-4db5-ba64-199b48d69eea`::ref 0 aborting False >Thread-77::DEBUG::2014-07-14 17:53:18,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:53:18,158::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000611764 s, 606 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:53:18,171::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:53:18,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:19,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542556 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:21,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:21,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913356 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:53:21,453::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:53:21,466::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:53:21,468::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:53:21,470::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:53:21,472::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:53:21,472::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:53:21,472::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:53:21,473::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:53:21,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673479 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:23,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:23,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914337 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:25,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:25,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916821 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:53:25,769::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:53:25,770::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:53:27,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:27,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913151 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:53:28,161::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:53:28,174::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00062288 s, 596 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:29,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:29,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912393 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:53:30,392::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:53:31,489::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:53:31,501::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000736155 s, 710 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:31,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:31,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912313 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:53:33,184::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:53:33,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:33,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912813 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:53:35,773::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:53:35,774::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:53:35,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:36,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507258 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:53:38,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:53:38,188::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000613421 s, 605 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:38,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:38,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912447 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:40,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:53:42,105::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:53:42,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000721801 s, 725 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:42,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.412 s, 725 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:44,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:44,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914104 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:53:45,794::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:53:45,797::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:53:46,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:46,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100175 s, 10.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:53:48,190::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-113::DEBUG::2014-07-14 17:53:48,201::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:53:48,207::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:53:48,208::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:53:48,211::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:53:48,213::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:53:48,213::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:53:48,213::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:53:48,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:53:48,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649948 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:48,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:49,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542813 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:51,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:51,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132892 s, 7.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:53:52,125::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:53:52,137::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689633 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:53,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:53,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546667 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:53:55,804::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:53:55,805::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:53:55,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:55,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912766 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:53:57,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:53:58,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913413 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:53:58,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:53:58,241::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680135 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:00,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:00,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912562 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:54:00,424::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:54:02,141::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:54:02,153::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069194 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:02,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:02,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09118 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:54:03,215::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:54:04,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:04,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912389 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:54:05,816::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:54:05,817::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:54:06,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:06,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914109 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:54:08,244::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:54:08,256::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649738 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:08,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:08,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913285 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:10,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:10,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913146 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:54:12,157::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:54:12,168::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000712484 s, 734 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:12,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:12,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919159 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:14,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:14,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914092 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:54:15,828::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:54:15,832::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:54:16,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:17,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912504 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:54:18,218::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a2a3c0a-4588-4975-8683-80ebd8389ec4`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:54:18,218::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:54:18,221::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:54:18,222::task::1191::Storage.TaskManager.Task::(prepare) Task=`4a2a3c0a-4588-4975-8683-80ebd8389ec4`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:54:18,222::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a2a3c0a-4588-4975-8683-80ebd8389ec4`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:54:18,222::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:54:18,222::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:54:18,222::task::993::Storage.TaskManager.Task::(_decref) Task=`4a2a3c0a-4588-4975-8683-80ebd8389ec4`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:54:18,249::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:54:18,258::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:54:18,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006783 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:19,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:19,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912261 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:21,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:21,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911434 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:54:22,172::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:54:22,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697903 s, 749 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:23,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:23,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911513 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:25,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:25,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912378 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:54:25,839::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:54:25,840::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:54:27,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:27,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913474 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:54:28,273::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:54:28,286::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000614574 s, 604 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:29,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:29,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911543 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:54:30,465::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:54:31,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:31,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913163 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:54:32,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:54:32,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696674 s, 751 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:54:33,261::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:54:33,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:34,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911487 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:54:35,852::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:54:35,853::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:54:36,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:36,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.449061 s, 2.3 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:54:38,289::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:54:38,301::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627646 s, 591 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:38,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:38,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913387 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:40,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:40,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914296 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:54:42,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:54:42,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693645 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:42,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:42,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911326 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:45,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:45,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916974 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:54:45,869::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:54:45,872::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:54:47,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:47,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913683 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:54:48,274::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:54:48,305::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:54:48,316::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692529 s, 536 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:49,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:49,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091268 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:51,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:51,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912024 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:54:52,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:54:52,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695646 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:53,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:53,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911254 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:54:55,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:55,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132937 s, 7.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:54:55,879::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:54:55,880::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:54:57,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:54:58,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551031 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:54:58,322::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:54:58,335::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686049 s, 541 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:00,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:00,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911913 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:55:00,498::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-29::DEBUG::2014-07-14 17:55:02,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:55:02,246::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668556 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:02,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:02,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912812 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:55:03,286::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:55:04,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:04,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912473 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:55:05,891::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:55:05,892::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:55:06,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:06,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914598 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:55:08,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:55:08,352::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664812 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:08,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:08,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913069 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:10,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:11,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559476 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:55:12,250::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:55:12,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00080616 s, 649 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:13,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:13,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911535 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:15,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:55:15,906::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:55:15,911::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:55:16,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551971 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:55:18,289::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4bd6ad8-9f57-4ada-b750-24a7a95a9fc7`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:55:18,290::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:55:18,293::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:55:18,293::task::1191::Storage.TaskManager.Task::(prepare) Task=`e4bd6ad8-9f57-4ada-b750-24a7a95a9fc7`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:55:18,293::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4bd6ad8-9f57-4ada-b750-24a7a95a9fc7`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:55:18,293::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:55:18,293::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:55:18,294::task::993::Storage.TaskManager.Task::(_decref) Task=`e4bd6ad8-9f57-4ada-b750-24a7a95a9fc7`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:55:18,321::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:55:18,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:55:18,356::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:55:18,446::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635374 s, 584 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:18,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0961045 s, 10.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:20,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:20,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0940438 s, 10.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:55:22,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:55:22,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000717008 s, 729 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:22,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:22,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913524 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:24,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:24,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091168 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:55:25,916::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:55:25,917::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:55:26,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:26,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911721 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:55:28,452::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:55:28,465::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649963 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:28,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:29,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913011 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:55:30,529::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:55:31,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:31,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551119 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:55:32,412::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:55:32,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689447 s, 759 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:55:33,333::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:55:33,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:33,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911497 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:35,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:35,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912094 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:55:35,928::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:55:35,930::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:55:37,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:37,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09125 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:55:38,468::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:55:38,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645528 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:39,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:40,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:42,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:42,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912178 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:55:42,427::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:55:42,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681297 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:44,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:44,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912018 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:55:45,944::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:55:45,947::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:55:46,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:46,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967073 s, 10.6 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:55:48,346::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:55:48,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:55:48,494::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677661 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:48,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:48,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911271 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:50,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:50,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913676 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:55:52,443::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:55:52,455::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689868 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:52,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:53,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545863 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:55,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:55,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912019 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:55:55,951::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:55:55,952::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:55:57,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:57,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919195 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:55:58,497::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:55:58,509::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067779 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:55:59,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:55:59,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.133645 s, 7.7 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:56:00,561::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:56:01,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:01,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912272 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:56:02,459::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:56:02,471::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694259 s, 753 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:56:03,358::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:56:03,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:03,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912404 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:56:05,963::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:56:05,964::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:56:06,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:06,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913272 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:08,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:08,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912173 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:56:08,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:56:08,524::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683451 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:10,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:10,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913348 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:12,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:12,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914562 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:56:12,475::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:56:12,486::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692037 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:14,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:14,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0939233 s, 10.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:56:15,978::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:56:15,982::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:56:16,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:16,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911615 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:56:18,364::task::595::Storage.TaskManager.Task::(_updateState) Task=`bad781ec-3905-47e9-a785-28f1b6b0ee9c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:56:18,365::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:56:18,367::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:56:18,368::task::1191::Storage.TaskManager.Task::(prepare) Task=`bad781ec-3905-47e9-a785-28f1b6b0ee9c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:56:18,368::task::595::Storage.TaskManager.Task::(_updateState) Task=`bad781ec-3905-47e9-a785-28f1b6b0ee9c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:56:18,368::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:56:18,368::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:56:18,368::task::993::Storage.TaskManager.Task::(_decref) Task=`bad781ec-3905-47e9-a785-28f1b6b0ee9c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:56:18,395::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:56:18,527::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:56:18,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645775 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:18,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:18,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911493 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:20,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:20,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911299 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:56:22,490::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:56:22,501::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680281 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:22,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:23,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910681 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:25,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:25,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920652 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:56:25,988::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:56:25,989::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:56:27,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:27,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912496 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:56:28,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:56:28,554::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000640659 s, 579 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:29,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:29,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912482 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:56:30,594::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:56:31,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:31,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911757 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:56:32,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:56:32,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680575 s, 768 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:56:33,408::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:56:33,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:33,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911795 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:35,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:35,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913001 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:56:36,000::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:56:36,002::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:56:37,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:37,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914076 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:56:38,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:56:38,570::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656256 s, 565 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:39,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:39,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914718 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:41,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:42,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912965 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:56:42,521::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:56:42,532::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000651676 s, 803 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:44,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:44,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919074 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:56:46,016::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:56:46,019::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:56:46,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:46,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551913 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:56:48,421::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:56:48,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:56:48,588::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660367 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:48,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:48,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911038 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:50,926::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:51,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911554 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:56:52,536::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:56:52,548::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677119 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:53,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:53,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913489 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:55,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:55,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927439 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:56:56,027::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:56:56,028::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:56:57,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:57,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913263 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:56:58,592::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:56:58,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065062 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:56:59,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:56:59,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913159 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:57:00,626::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:57:01,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:01,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914155 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:57:02,552::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:57:02,564::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689065 s, 759 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:57:03,434::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:57:03,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:03,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913004 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:05,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:57:06,035::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:57:06,037::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:57:06,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544831 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:08,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:57:08,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:08,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55396 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:57:08,921::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00599627 s, 61.9 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:10,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:11,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.133291 s, 7.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:57:12,568::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:57:12,579::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683274 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:13,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:13,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913891 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:15,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:15,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913345 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:57:16,045::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:57:16,049::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:57:17,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:17,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091226 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:57:18,438::task::595::Storage.TaskManager.Task::(_updateState) Task=`042a7ab9-cef8-42c5-8c9f-fdc87bcc3607`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:57:18,438::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:57:18,441::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:57:18,441::task::1191::Storage.TaskManager.Task::(prepare) Task=`042a7ab9-cef8-42c5-8c9f-fdc87bcc3607`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:57:18,441::task::595::Storage.TaskManager.Task::(_updateState) Task=`042a7ab9-cef8-42c5-8c9f-fdc87bcc3607`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:57:18,442::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:57:18,442::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:57:18,442::task::993::Storage.TaskManager.Task::(_decref) Task=`042a7ab9-cef8-42c5-8c9f-fdc87bcc3607`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:57:18,468::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:57:18,930::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:57:18,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000651074 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:19,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:19,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542807 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:22,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:22,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091309 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:57:22,583::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:57:22,595::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000699469 s, 748 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:24,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:24,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912824 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:57:26,056::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:57:26,057::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:57:26,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:26,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970719 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:28,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:28,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913507 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:57:28,945::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:57:28,957::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000611411 s, 607 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:30,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:57:30,657::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:57:31,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54177 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:57:32,599::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:57:32,611::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000633651 s, 825 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:33,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:57:33,479::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:57:33,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541676 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:35,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:35,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913456 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:57:36,065::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:57:36,066::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:57:37,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:37,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912554 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:57:38,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:57:38,973::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000700355 s, 530 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:39,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:39,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913676 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:41,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:42,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911799 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:57:42,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:57:42,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00074038 s, 706 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:44,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:44,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912951 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:57:46,083::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:57:46,088::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:57:46,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:46,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100181 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:48,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:57:48,492::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:57:48,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912607 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:57:48,976::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:57:48,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669832 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:50,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:50,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913776 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:57:52,630::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:57:52,642::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000652197 s, 802 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:52,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:52,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911055 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:54,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:55,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.513977 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:57:56,091::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:57:56,092::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:57:57,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:57,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0931083 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:57:58,991::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:57:59,002::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685378 s, 541 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:57:59,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:57:59,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0936183 s, 10.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:58:00,689::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:58:01,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:01,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912109 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:58:02,646::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:58:02,657::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674332 s, 776 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:58:03,504::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:58:03,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:04,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554023 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:58:06,103::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:58:06,104::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:58:06,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:06,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912856 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:08,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:08,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912044 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:58:09,006::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:58:09,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000739232 s, 502 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:10,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:10,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091395 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:12,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:58:12,744::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:58:12,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000796893 s, 656 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:12,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104773 s, 9.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:14,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:14,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132474 s, 7.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:58:16,129::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:58:16,132::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:58:16,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:17,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912935 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:58:18,507::task::595::Storage.TaskManager.Task::(_updateState) Task=`10340013-4ce7-4bd9-8a0b-4cf290e50aa4`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:58:18,507::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:58:18,510::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:58:18,511::task::1191::Storage.TaskManager.Task::(prepare) Task=`10340013-4ce7-4bd9-8a0b-4cf290e50aa4`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:58:18,511::task::595::Storage.TaskManager.Task::(_updateState) Task=`10340013-4ce7-4bd9-8a0b-4cf290e50aa4`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:58:18,511::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:58:18,511::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:58:18,511::task::993::Storage.TaskManager.Task::(_decref) Task=`10340013-4ce7-4bd9-8a0b-4cf290e50aa4`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 17:58:18,537::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:58:19,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:58:19,032::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690086 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:19,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:19,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913249 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:21,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:21,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913828 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:58:22,764::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:58:22,777::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 17:58:22,778::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 17:58:22,781::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 17:58:22,783::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 17:58:22,783::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 17:58:22,783::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 17:58:22,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:58:22,796::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068792 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:23,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:23,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55886 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:25,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:25,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912491 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:58:26,135::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:58:26,136::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:58:27,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:28,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913919 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:58:29,037::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:58:29,049::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643895 s, 576 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:30,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:30,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912817 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:58:30,723::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:58:32,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:32,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912181 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:58:32,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:58:32,811::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067789 s, 772 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:58:33,550::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:58:34,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:34,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912405 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:58:36,143::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:58:36,144::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:58:36,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:36,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914399 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:38,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 17:58:39,106::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:58:39,119::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000719123 s, 516 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:39,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.530064 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:41,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:41,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913712 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:58:42,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:58:42,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700106 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:43,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:43,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558709 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:45,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:45,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915467 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:58:46,161::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:58:46,167::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:58:48,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:48,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913115 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:58:48,562::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:58:49,124::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:58:49,138::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 17:58:49,140::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 17:58:49,142::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 17:58:49,144::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 17:58:49,144::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 17:58:49,144::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 17:58:49,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:58:49,157::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664509 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:50,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:50,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09123 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:52,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:52,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091335 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:58:52,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:58:52,843::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657116 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:54,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:54,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912607 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:58:56,166::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:58:56,168::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:58:56,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:56,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091331 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:58:58,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:58:58,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913058 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:58:59,161::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:58:59,172::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642028 s, 578 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:00,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 17:59:00,755::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:59:00,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913581 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:59:02,847::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:59:02,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000700946 s, 746 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:02,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:02,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912202 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:59:03,575::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:59:04,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:05,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912937 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:59:06,178::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:59:06,179::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:59:07,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:07,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914047 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:59:09,176::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:59:09,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693652 s, 535 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:09,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:09,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913239 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:11,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:11,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921589 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:59:12,863::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:59:12,874::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630046 s, 830 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:13,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:14,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543094 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:16,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:59:16,193::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:59:16,196::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:59:16,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54236 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:59:18,578::task::595::Storage.TaskManager.Task::(_updateState) Task=`d80338de-8884-4e43-9348-da875864fec6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 17:59:18,579::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 17:59:18,582::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:59:18,582::task::1191::Storage.TaskManager.Task::(prepare) Task=`d80338de-8884-4e43-9348-da875864fec6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 17:59:18,582::task::595::Storage.TaskManager.Task::(_updateState) Task=`d80338de-8884-4e43-9348-da875864fec6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 17:59:18,582::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 17:59:18,583::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 17:59:18,583::task::993::Storage.TaskManager.Task::(_decref) Task=`d80338de-8884-4e43-9348-da875864fec6`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 17:59:18,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 17:59:18,621::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:59:18,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910669 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:59:19,191::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:59:19,202::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689099 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:20,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:20,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912256 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:22,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:59:22,924::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:59:22,938::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000648627 s, 806 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:22,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108364 s, 9.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:24,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:25,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911392 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:59:26,203::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:59:26,205::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:59:27,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:27,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560466 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:59:29,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:59:29,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664859 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:29,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:29,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.13457 s, 7.6 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 17:59:30,788::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 17:59:31,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:31,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915573 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:59:32,949::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:59:32,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677195 s, 772 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:59:33,633::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 17:59:33,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:34,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913277 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:36,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 17:59:36,212::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:59:36,214::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:59:36,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.56077 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:38,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:38,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914013 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:59:39,221::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:59:39,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065434 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:40,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:40,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913139 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:42,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 17:59:42,986::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:59:42,999::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000782201 s, 669 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:43,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.107813 s, 9.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:45,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:45,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913651 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:59:46,228::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:59:46,231::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:59:47,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:47,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911053 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 17:59:48,645::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 17:59:49,237::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:59:49,248::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682885 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:49,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:49,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912373 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:51,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:51,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913446 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 17:59:53,009::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 17:59:53,021::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064234 s, 814 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:53,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:53,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913827 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 17:59:55,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:56,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550315 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 17:59:56,238::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 17:59:56,239::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 17:59:58,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 17:59:58,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912806 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 17:59:59,252::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 17:59:59,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000633748 s, 585 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:00,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:00,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912588 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:00:00,821::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:00:02,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:02,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912225 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:00:03,026::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:00:03,038::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650028 s, 805 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:00:03,657::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:00:04,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:04,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911822 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:00:06,246::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:00:06,247::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:00:06,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:07,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911665 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:00:09,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:00:09,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671631 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:09,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:09,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914929 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:11,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:12,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542714 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:00:13,042::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:00:13,053::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676187 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:14,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:14,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912339 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:00:16,261::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:00:16,264::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:00:16,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:16,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543748 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:00:18,668::task::595::Storage.TaskManager.Task::(_updateState) Task=`357f42f4-4728-4611-ace0-77d26df9e083`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:00:18,668::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:00:18,671::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:00:18,671::task::1191::Storage.TaskManager.Task::(prepare) Task=`357f42f4-4728-4611-ace0-77d26df9e083`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:00:18,671::task::595::Storage.TaskManager.Task::(_updateState) Task=`357f42f4-4728-4611-ace0-77d26df9e083`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:00:18,672::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:00:18,672::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:00:18,672::task::993::Storage.TaskManager.Task::(_decref) Task=`357f42f4-4728-4611-ace0-77d26df9e083`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:00:18,699::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:00:18,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:00:20,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:20,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.38634 s, 739 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:00:20,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000778928 s, 476 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:22,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:22,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968838 s, 10.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:00:23,057::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:00:23,069::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000631309 s, 828 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:24,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:24,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912723 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:00:26,271::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:00:26,272::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:00:26,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:26,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912733 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:28,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:28,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912428 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:00:30,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:00:30,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617907 s, 600 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:30,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 18:00:30,855::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:00:30,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912184 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:32,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:32,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912232 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:00:33,073::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:00:33,085::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661913 s, 790 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:00:33,712::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:00:35,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:35,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914809 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:00:36,279::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:00:36,280::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:00:37,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:37,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912397 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:39,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:39,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913105 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:00:40,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:00:40,330::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677956 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:41,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:41,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912862 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:00:43,089::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:00:43,101::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693297 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:43,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:43,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920071 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:45,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:45,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091651 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:00:46,294::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:00:46,298::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:00:47,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:47,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912904 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:00:48,723::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:00:49,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:50,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911683 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:00:50,333::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:00:50,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681584 s, 544 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:52,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:52,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091388 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:00:53,299::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:00:53,311::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000666083 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:54,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:54,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914895 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:56,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:00:56,312::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:00:56,313::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:00:56,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915757 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:00:58,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:00:58,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097457 s, 10.5 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:01:00,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:01:00,361::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674792 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:00,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:00,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913572 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:01:00,888::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:01:02,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:02,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091151 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:01:03,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:01:03,326::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636638 s, 822 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:01:03,744::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:01:04,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:04,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920537 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:01:06,310::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:01:06,312::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:01:06,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:06,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915151 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:09,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:09,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092965 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:01:10,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:01:10,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000671712 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:11,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:11,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918066 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:01:13,331::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:01:13,343::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000620217 s, 843 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:13,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:13,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912736 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:15,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:15,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970966 s, 10.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:01:16,322::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:01:16,326::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:01:17,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:18,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.110338 s, 9.3 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:01:18,747::task::595::Storage.TaskManager.Task::(_updateState) Task=`47e44a9e-b144-4f47-b4cd-ac851ac18500`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:01:18,747::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:01:18,751::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:01:18,751::task::1191::Storage.TaskManager.Task::(prepare) Task=`47e44a9e-b144-4f47-b4cd-ac851ac18500`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:01:18,751::task::595::Storage.TaskManager.Task::(_updateState) Task=`47e44a9e-b144-4f47-b4cd-ac851ac18500`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:01:18,751::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:01:18,751::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:01:18,751::task::993::Storage.TaskManager.Task::(_decref) Task=`47e44a9e-b144-4f47-b4cd-ac851ac18500`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:01:18,777::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:01:20,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:20,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913126 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:01:20,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:01:20,391::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683244 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:22,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:22,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091327 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:01:23,347::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:01:23,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683219 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:24,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:24,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910705 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:01:26,331::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:01:26,333::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:01:26,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:26,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911783 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:28,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:28,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918063 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:01:30,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:01:30,408::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617396 s, 601 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:30,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:30,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913843 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:01:30,921::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:01:32,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:32,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912612 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:01:33,363::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:01:33,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653622 s, 800 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:01:33,789::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:01:34,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:35,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543753 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:01:36,343::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:01:36,345::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:01:37,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:37,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919398 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:39,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:39,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912877 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:01:40,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:01:40,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065601 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:41,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:41,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912636 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:01:43,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:01:43,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000661114 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:43,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:43,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911281 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:45,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:01:46,359::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:01:46,363::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:01:46,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54495 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:48,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:01:48,801::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:01:49,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547018 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:01:50,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:01:50,438::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677793 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:51,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:51,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912438 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:53,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:53,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920565 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:01:53,396::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:01:53,408::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695399 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:55,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:55,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912476 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:01:56,369::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:01:56,370::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:01:57,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:57,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912904 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:01:59,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:01:59,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913023 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:02:00,441::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:02:00,453::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000684807 s, 542 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:02:00,949::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:02:01,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:01,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911305 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:02:03,412::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:02:03,424::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000739256 s, 707 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:02:03,813::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:02:03,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:03,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915025 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:05,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:06,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918762 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:02:06,381::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:02:06,382::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:02:08,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:08,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911738 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:10,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:10,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912743 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:02:10,456::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:02:10,468::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00070202 s, 528 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:12,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:12,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912162 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:02:13,428::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:02:13,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000672711 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:14,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:14,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912601 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:02:16,403::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:02:16,406::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:02:16,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:16,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912008 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:18,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:18,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912459 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:02:18,816::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d04adba-177f-4fb9-b251-5ba81feabac1`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:02:18,817::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:02:18,820::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:02:18,820::task::1191::Storage.TaskManager.Task::(prepare) Task=`6d04adba-177f-4fb9-b251-5ba81feabac1`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:02:18,820::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d04adba-177f-4fb9-b251-5ba81feabac1`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:02:18,820::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:02:18,820::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:02:18,821::task::993::Storage.TaskManager.Task::(_decref) Task=`6d04adba-177f-4fb9-b251-5ba81feabac1`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:02:18,847::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 18:02:20,471::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:02:20,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654374 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:20,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:20,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912198 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:22,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:23,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912495 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:02:23,443::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:02:23,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686429 s, 762 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:25,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:25,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913287 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:02:26,413::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:02:26,414::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:02:27,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:27,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912281 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:29,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:29,377::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912781 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:02:30,487::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:02:30,499::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655149 s, 566 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:02:30,983::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:02:31,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:31,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912424 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:02:33,458::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:02:33,470::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662184 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:33,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:33,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912697 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:02:33,860::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:02:35,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:35,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091338 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:02:36,419::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:02:36,420::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:02:37,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:37,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912129 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:39,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:39,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914044 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:02:40,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:02:40,514::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660383 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:41,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:42,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092402 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:02:43,474::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:02:43,485::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000740486 s, 706 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:44,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:44,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912324 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:46,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:02:46,442::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:02:46,447::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:02:46,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.56041 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:48,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:02:48,873::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:02:48,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913089 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:02:50,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:02:50,529::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067721 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:50,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:51,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911883 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:53,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:02:53,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:02:53,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000854896 s, 612 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:53,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.580786 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:55,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:55,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919697 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:02:56,450::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:02:56,451::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:02:57,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:57,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911808 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:02:59,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:02:59,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911139 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:03:00,532::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:03:00,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669463 s, 554 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:03:01,016::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:03:01,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:02,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912667 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:03:03,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:03:03,638::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000881862 s, 593 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:03:03,886::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:03:04,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:04,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911018 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:06,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:06,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919118 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:03:06,462::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:03:06,464::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:03:08,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:08,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912746 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:10,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:03:10,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:10,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102907 s, 10.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:03:10,580::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000753683 s, 492 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:12,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:13,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559841 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:03:13,642::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:03:13,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0012484 s, 419 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:15,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:15,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.567429 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:03:16,480::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:03:16,485::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:03:17,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:18,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547793 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:03:18,889::task::595::Storage.TaskManager.Task::(_updateState) Task=`39167836-8b65-4faf-aced-3b7e2d0c565c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:03:18,889::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:03:18,892::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:03:18,892::task::1191::Storage.TaskManager.Task::(prepare) Task=`39167836-8b65-4faf-aced-3b7e2d0c565c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:03:18,893::task::595::Storage.TaskManager.Task::(_updateState) Task=`39167836-8b65-4faf-aced-3b7e2d0c565c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:03:18,893::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:03:18,893::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:03:18,893::task::993::Storage.TaskManager.Task::(_decref) Task=`39167836-8b65-4faf-aced-3b7e2d0c565c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:03:18,919::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:03:20,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:20,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911638 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:03:20,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:03:20,599::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000716067 s, 518 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:22,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:22,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911014 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:03:23,656::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:03:23,670::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:03:23,671::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 18:03:23,674::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 18:03:23,675::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 18:03:23,675::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 18:03:23,676::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 18:03:23,677::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:03:23,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000609621 s, 858 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:24,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:24,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912183 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:03:26,488::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:03:26,489::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:03:26,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:26,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091265 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:28,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:29,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910715 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:03:30,606::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:03:30,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627027 s, 592 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:03:31,048::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:03:31,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:31,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561056 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:33,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:03:33,695::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:33,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917228 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:03:33,874::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.089189 s, 5.9 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:03:33,931::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:03:35,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:35,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912451 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:03:36,495::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:03:36,496::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:03:37,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:38,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554564 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:40,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:40,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914753 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:03:40,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:03:40,634::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006722 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:42,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:42,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912391 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:03:43,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:03:43,889::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676208 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:44,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:44,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091454 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:03:46,513::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:03:46,518::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:03:46,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:46,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912471 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:03:48,942::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:03:48,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:49,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914716 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:03:50,636::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:03:50,650::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:03:50,651::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 18:03:50,654::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 18:03:50,655::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 18:03:50,655::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 18:03:50,656::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 18:03:50,657::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:03:50,668::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686892 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:51,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:51,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549915 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:53,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:03:54,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:03:54,224::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00155963 s, 335 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:54,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551467 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:03:56,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:56,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925243 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:03:56,521::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:03:56,522::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:03:58,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:03:58,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912405 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:00,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:00,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914308 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:04:00,672::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:04:00,683::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690296 s, 537 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:04:01,081::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:04:02,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:02,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913153 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:04:03,954::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:04:04,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:04:04,242::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00071432 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:04,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:04,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911422 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:04:06,535::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:04:06,539::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:04:06,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:06,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091434 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:08,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:09,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911887 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:04:10,686::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:04:10,697::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000633314 s, 586 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:11,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:11,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912074 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:13,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:13,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911426 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:04:14,246::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:04:14,257::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674648 s, 775 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:15,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:15,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967913 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:04:16,551::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:04:16,557::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:04:17,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:17,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913833 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:04:18,958::task::595::Storage.TaskManager.Task::(_updateState) Task=`a35cd8c0-6ce9-4086-bd78-ec7560f0cf49`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:04:18,958::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:04:18,961::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:04:18,962::task::1191::Storage.TaskManager.Task::(prepare) Task=`a35cd8c0-6ce9-4086-bd78-ec7560f0cf49`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:04:18,962::task::595::Storage.TaskManager.Task::(_updateState) Task=`a35cd8c0-6ce9-4086-bd78-ec7560f0cf49`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:04:18,962::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:04:18,962::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:04:18,962::task::993::Storage.TaskManager.Task::(_decref) Task=`a35cd8c0-6ce9-4086-bd78-ec7560f0cf49`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:04:18,990::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:04:19,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:19,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913084 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:04:20,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:04:20,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066798 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:21,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:22,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911906 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:24,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:24,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911773 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:04:24,261::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:04:24,272::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688678 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:26,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:26,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912389 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:04:26,559::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:04:26,561::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:04:28,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:28,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912337 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:30,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:30,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912504 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:04:30,716::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:04:30,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000618452 s, 600 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:04:31,114::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:04:32,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:32,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911832 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:04:34,002::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:04:34,276::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:04:34,288::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632608 s, 827 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:34,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:34,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912241 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:04:36,570::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:04:36,571::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:04:36,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:37,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515903 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:39,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:39,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913115 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:04:40,731::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:04:40,743::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672765 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:41,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:41,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912491 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:43,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:43,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911586 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:04:44,291::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:04:44,303::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000708588 s, 738 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:45,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:45,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0965083 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:04:46,583::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:04:46,584::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:04:47,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:48,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911867 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:04:49,014::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:04:50,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:50,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913543 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:04:50,746::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:04:50,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690783 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:52,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:52,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911444 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:54,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:04:54,309::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:04:54,399::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665093 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:54,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0977977 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:04:56,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:56,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915079 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:04:56,590::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:04:56,592::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:04:58,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:04:58,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913908 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:00,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:00,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916046 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:05:00,763::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:05:00,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00566616 s, 65.5 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:05:01,146::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:05:02,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:02,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913346 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:05:04,026::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:05:04,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:05:04,418::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621194 s, 842 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:04,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:05,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0955728 s, 10.7 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:05:06,600::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:05:06,601::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:05:07,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:07,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913573 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:09,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:09,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912397 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:05:10,786::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:05:10,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652061 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:11,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:11,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914491 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:13,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:13,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915029 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:05:14,422::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:05:14,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689337 s, 759 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:15,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:15,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0968343 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:05:16,611::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:05:16,612::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:05:17,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:17,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913512 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:05:19,029::task::595::Storage.TaskManager.Task::(_updateState) Task=`f72b032e-1418-4a41-9e1e-4b409b492fb8`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:05:19,029::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:05:19,033::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:05:19,033::task::1191::Storage.TaskManager.Task::(prepare) Task=`f72b032e-1418-4a41-9e1e-4b409b492fb8`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:05:19,033::task::595::Storage.TaskManager.Task::(_updateState) Task=`f72b032e-1418-4a41-9e1e-4b409b492fb8`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:05:19,033::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:05:19,033::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:05:19,033::task::993::Storage.TaskManager.Task::(_decref) Task=`f72b032e-1418-4a41-9e1e-4b409b492fb8`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:05:19,061::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:05:19,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:19,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913126 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:05:20,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:05:20,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000685676 s, 541 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:21,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:22,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915599 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:24,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:24,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912839 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:05:24,438::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:05:24,449::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679842 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:26,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:05:26,622::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:05:26,623::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:05:26,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550737 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:28,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:28,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912511 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:05:30,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:05:30,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646615 s, 574 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:30,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:31,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913268 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:05:31,184::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:05:33,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:33,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912372 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:05:34,073::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:05:34,453::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:05:34,465::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660828 s, 791 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:35,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:35,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911925 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:05:36,631::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:05:36,633::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:05:37,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:37,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915686 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:39,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:39,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912514 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:05:40,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:05:40,843::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000629854 s, 589 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:41,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:41,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913045 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:43,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:43,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914021 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:05:44,469::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:05:44,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703776 s, 743 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:45,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:45,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100045 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:05:46,644::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:05:46,645::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:05:47,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:48,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920307 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:05:49,086::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:05:50,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:50,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091142 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:05:50,846::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:05:50,858::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676982 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:52,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:52,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551102 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:05:54,484::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:05:54,496::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692916 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:54,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:54,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913087 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:05:56,651::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:05:56,653::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:05:56,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:57,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914559 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:05:59,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:05:59,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921397 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:06:00,861::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:06:00,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000616166 s, 602 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:01,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 18:06:01,218::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:06:01,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:03,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:03,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911081 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:06:04,097::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:06:04,499::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:06:04,511::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691833 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:05,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:05,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.097646 s, 10.5 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:06:06,664::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:06:06,665::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:06:07,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:07,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912947 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:09,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:09,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091319 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:06:10,876::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:06:10,888::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000704735 s, 526 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:11,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:12,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551875 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:06:14,514::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:06:14,526::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681151 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:14,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:14,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918249 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:16,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:06:16,672::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:06:16,674::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:06:16,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923097 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:18,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:18,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925869 s, 11.1 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:06:19,100::task::595::Storage.TaskManager.Task::(_updateState) Task=`5108a948-b3a4-44ef-a5eb-f2ae87b8fd93`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:06:19,101::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:06:19,103::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:06:19,104::task::1191::Storage.TaskManager.Task::(prepare) Task=`5108a948-b3a4-44ef-a5eb-f2ae87b8fd93`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:06:19,104::task::595::Storage.TaskManager.Task::(_updateState) Task=`5108a948-b3a4-44ef-a5eb-f2ae87b8fd93`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:06:19,104::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:06:19,104::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:06:19,104::task::993::Storage.TaskManager.Task::(_decref) Task=`5108a948-b3a4-44ef-a5eb-f2ae87b8fd93`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:06:19,131::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::DEBUG::2014-07-14 18:06:20,891::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:20,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:06:20,916::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000734285 s, 505 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:21,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915168 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:23,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:23,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559636 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:06:24,530::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:06:24,541::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692717 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:25,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:25,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912073 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:06:26,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:06:26,683::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:06:27,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:27,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912498 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:29,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:30,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543533 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:06:30,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:06:30,932::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000606665 s, 612 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:06:31,250::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:06:32,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:32,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912838 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:06:34,144::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:06:34,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:06:34,632::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:06:34,645::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657739 s, 795 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:34,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.111377 s, 9.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:36,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:06:36,695::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:06:36,697::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:06:36,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912595 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:38,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:38,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912133 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:40,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:06:41,010::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:41,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104488 s, 9.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:06:41,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00444146 s, 83.5 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:43,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:43,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918205 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:06:44,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:06:44,666::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697027 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:45,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:45,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091393 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:06:46,707::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:06:46,708::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:06:47,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:47,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091249 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:06:49,156::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:06:49,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:49,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913112 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:06:51,034::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:06:51,046::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691232 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:51,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:52,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558767 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:54,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:54,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912394 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:06:54,670::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:06:54,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00064213 s, 814 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:56,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:06:56,720::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:06:56,721::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:06:56,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544379 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:06:58,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:06:58,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912711 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:00,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:01,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912818 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:07:01,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:07:01,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000681764 s, 544 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:07:01,283::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:07:03,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:03,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915899 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:07:04,169::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:07:04,685::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:07:04,697::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068669 s, 762 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:05,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:05,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913808 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:07:06,732::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:07:06,733::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:07:07,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:07,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132284 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:09,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:09,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546795 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:07:11,064::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:07:11,076::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683045 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:11,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:12,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913788 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:14,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:14,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912385 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:07:14,701::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:07:14,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067703 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:16,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:16,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0988502 s, 10.4 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:07:16,742::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:07:16,743::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:07:18,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:18,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914458 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:07:19,175::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c15f078-ad2e-46a8-8358-2820f18201b1`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:07:19,176::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:07:19,179::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:07:19,179::task::1191::Storage.TaskManager.Task::(prepare) Task=`0c15f078-ad2e-46a8-8358-2820f18201b1`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:07:19,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c15f078-ad2e-46a8-8358-2820f18201b1`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:07:19,179::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:07:19,179::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:07:19,180::task::993::Storage.TaskManager.Task::(_decref) Task=`0c15f078-ad2e-46a8-8358-2820f18201b1`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:07:19,205::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:07:20,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:20,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912059 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:07:21,079::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:07:21,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000655454 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:22,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:22,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100674 s, 10.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:07:24,716::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:07:24,728::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000702816 s, 744 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:24,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:25,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938695 s, 10.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:07:26,754::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:07:26,755::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:07:27,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:27,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912856 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:29,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:29,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925233 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:07:31,095::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:07:31,106::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648575 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:31,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 18:07:31,316::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:07:31,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550813 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:33,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:33,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913908 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:07:34,216::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:07:34,732::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:07:34,744::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00067269 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:35,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:36,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914805 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:07:36,762::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:07:36,764::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:07:38,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:38,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914359 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:40,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:40,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914474 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:07:41,110::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:07:41,122::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065866 s, 563 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:42,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:42,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913201 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:44,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:44,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091985 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:07:44,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:07:44,758::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643874 s, 812 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:46,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:46,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912644 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:07:46,773::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:07:46,774::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:07:48,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:48,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913808 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:07:49,227::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:07:50,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:50,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091392 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:07:51,125::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:07:51,137::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000640551 s, 579 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:52,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:53,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558951 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:07:54,762::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:07:54,774::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000636656 s, 821 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:55,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:55,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913084 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:07:56,782::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:07:56,783::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:07:57,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:07:57,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914096 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:07:59,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:00,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550744 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:08:01,140::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:08:01,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693304 s, 535 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:08:01,347::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:08:02,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:02,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.56066 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:08:04,239::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:08:04,778::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:08:04,790::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000741021 s, 706 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:04,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:06,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37571 s, 744 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:08:06,795::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:08:06,796::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:08:08,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:08,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912874 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:10,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:10,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915124 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:08:11,155::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:08:11,172::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068524 s, 541 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:12,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:12,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914054 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:14,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:14,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913527 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:08:14,793::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:08:14,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687616 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:16,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:08:16,812::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:08:16,814::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:08:16,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0978048 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:18,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:19,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913067 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:08:19,241::task::595::Storage.TaskManager.Task::(_updateState) Task=`683cd721-94e7-45fc-bc6c-cde4b0f61959`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:08:19,242::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:08:19,245::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:08:19,245::task::1191::Storage.TaskManager.Task::(prepare) Task=`683cd721-94e7-45fc-bc6c-cde4b0f61959`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:08:19,245::task::595::Storage.TaskManager.Task::(_updateState) Task=`683cd721-94e7-45fc-bc6c-cde4b0f61959`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:08:19,245::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:08:19,245::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:08:19,245::task::993::Storage.TaskManager.Task::(_decref) Task=`683cd721-94e7-45fc-bc6c-cde4b0f61959`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:08:19,273::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:08:21,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:21,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091401 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:08:21,175::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:08:21,187::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667821 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:23,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:23,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913606 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:08:24,808::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:08:24,821::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:08:24,822::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 18:08:24,825::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 18:08:24,826::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 18:08:24,826::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 18:08:24,827::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 18:08:24,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:08:24,839::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643285 s, 813 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:25,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:25,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912154 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:08:26,822::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:08:26,823::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:08:27,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:27,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915403 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:29,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:30,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541576 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:08:31,190::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:08:31,202::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648537 s, 572 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:08:31,380::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:08:32,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:32,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913838 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:34,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:08:34,293::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:08:34,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921809 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:08:34,843::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:08:34,855::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065742 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:36,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:36,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914874 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:08:36,834::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:08:36,836::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:08:38,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:38,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913395 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:40,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:40,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911796 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:08:41,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:08:41,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669649 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:42,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:42,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914409 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:44,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:08:44,877::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:44,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103888 s, 9.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:08:44,892::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00226377 s, 231 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:08:46,842::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:08:46,843::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:08:46,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:47,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913301 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:49,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:49,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913217 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:08:49,306::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:08:51,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:08:51,219::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Dummy-49::DEBUG::2014-07-14 18:08:51,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104476 s, 9.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:08:51,294::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:08:51,296::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 18:08:51,299::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 18:08:51,303::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 18:08:51,304::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 18:08:51,304::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 18:08:51,309::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:08:51,322::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679442 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:53,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:53,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5449 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:08:54,902::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:08:54,914::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655558 s, 798 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:08:55,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:56,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091384 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:08:56,854::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:08:56,856::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:08:58,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:08:58,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544067 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:00,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:00,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09134 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:09:01,328::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:09:01,340::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00141916 s, 261 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:09:01,412::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:09:02,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:03,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54307 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:09:04,320::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:09:04,918::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:09:04,930::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000665457 s, 786 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:05,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:05,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545647 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:09:06,864::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:09:06,865::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:09:07,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:08,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543925 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:10,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:10,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913974 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:09:11,343::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:09:11,355::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000675695 s, 549 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:12,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:12,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913441 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:14,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:14,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914552 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:09:14,934::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:09:14,945::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000621739 s, 841 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:16,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:16,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913263 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:09:16,880::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:09:16,884::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:09:18,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:19,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913315 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:09:19,324::task::595::Storage.TaskManager.Task::(_updateState) Task=`faae7d7a-c912-47ce-a409-b6ed315dcde5`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:09:19,325::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:09:19,328::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:09:19,328::task::1191::Storage.TaskManager.Task::(prepare) Task=`faae7d7a-c912-47ce-a409-b6ed315dcde5`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:09:19,328::task::595::Storage.TaskManager.Task::(_updateState) Task=`faae7d7a-c912-47ce-a409-b6ed315dcde5`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:09:19,328::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:09:19,328::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:09:19,329::task::993::Storage.TaskManager.Task::(_decref) Task=`faae7d7a-c912-47ce-a409-b6ed315dcde5`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:09:19,357::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:09:21,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:21,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914248 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:09:21,358::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:09:21,372::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000634864 s, 584 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:23,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:23,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.555489 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:09:24,949::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:09:24,961::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687703 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:25,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:25,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911965 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:09:26,891::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:09:26,892::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:09:27,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:28,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550026 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:30,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:30,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547703 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:09:31,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:09:31,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000624481 s, 594 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:09:31,454::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:09:32,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:33,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913269 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:09:34,381::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:09:34,965::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:09:34,976::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000623336 s, 839 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:35,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:35,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911917 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:09:36,899::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:09:36,900::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:09:37,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:37,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913121 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:39,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:39,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912964 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:09:41,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:09:41,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654479 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:41,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:41,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911448 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:43,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:43,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914518 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:09:44,980::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:09:44,991::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683463 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:45,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:45,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967072 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:09:46,909::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:09:46,910::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:09:47,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:48,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914084 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:09:49,393::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:09:50,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:50,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913072 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:09:51,410::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:09:51,422::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680422 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:52,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:52,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912542 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:54,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:54,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.510649 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:09:54,996::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:09:55,008::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681336 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:56,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:09:56,923::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:09:56,928::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:09:56,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0999988 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:09:58,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:09:59,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914359 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:01,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:01,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918064 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:10:01,425::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:10:01,438::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000645522 s, 575 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:10:01,485::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:10:03,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:03,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911983 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:10:04,407::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:10:05,012::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:10:05,023::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068463 s, 764 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:05,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:05,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551335 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:10:06,932::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:10:06,933::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:10:07,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:07,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912694 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:09,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:10,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915136 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:10:11,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:10:11,932::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644801 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:12,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:12,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912533 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:14,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:14,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915848 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:10:15,027::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:10:15,039::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000737864 s, 709 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:16,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:16,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100064 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:10:16,943::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:10:16,944::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:10:18,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:18,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913713 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:10:19,410::task::595::Storage.TaskManager.Task::(_updateState) Task=`48e38d3d-afc2-4e8a-b7ca-dbc9b7831832`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:10:19,411::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:10:19,414::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:10:19,414::task::1191::Storage.TaskManager.Task::(prepare) Task=`48e38d3d-afc2-4e8a-b7ca-dbc9b7831832`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:10:19,414::task::595::Storage.TaskManager.Task::(_updateState) Task=`48e38d3d-afc2-4e8a-b7ca-dbc9b7831832`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:10:19,414::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:10:19,414::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:10:19,414::task::993::Storage.TaskManager.Task::(_decref) Task=`48e38d3d-afc2-4e8a-b7ca-dbc9b7831832`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:10:19,440::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:10:20,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:21,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.518023 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:10:21,935::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:10:21,948::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672034 s, 552 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:23,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:23,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558589 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:10:25,043::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:10:25,056::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00187991 s, 278 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:25,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:10:26,954::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:10:26,955::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:10:27,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37683 s, 744 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:29,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:29,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913765 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:31,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:31,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091258 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:10:31,517::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:10:31,952::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:10:31,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643174 s, 577 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:33,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:33,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0993409 s, 10.3 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:10:34,452::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:10:35,060::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:10:35,072::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677466 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:35,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:35,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914524 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:10:36,968::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:10:36,969::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:10:37,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:37,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915484 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:39,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:39,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915813 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:41,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:10:42,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:10:42,044::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000716438 s, 518 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:42,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103479 s, 9.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:44,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:44,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913921 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:10:45,076::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:10:45,087::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682537 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:46,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:46,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964599 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:10:46,977::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:10:46,978::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:10:48,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:48,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914047 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:10:49,464::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:10:50,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:50,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0940228 s, 10.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:10:52,050::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:10:52,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000633784 s, 585 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:52,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:53,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.354971 s, 2.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:55,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:10:55,146::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:55,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10335 s, 9.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:10:55,164::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00570602 s, 91.7 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:10:56,989::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:10:56,990::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:10:57,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:57,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.560618 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:10:59,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:10:59,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912195 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:11:01,551::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:11:01,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:01,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912626 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:11:02,065::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:11:02,077::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000693006 s, 535 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:04,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:04,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917331 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:11:04,475::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:11:05,172::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:11:05,184::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683508 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:06,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:06,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913259 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:11:07,002::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:11:07,003::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:11:08,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:08,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091529 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:10,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:10,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917651 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:11:12,080::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:11:12,092::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000698699 s, 531 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:12,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:12,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092281 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:14,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:14,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914635 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:11:15,188::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:11:15,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000667893 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:16,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:16,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913289 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:11:17,009::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:11:17,010::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:11:18,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:18,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912613 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:11:19,478::task::595::Storage.TaskManager.Task::(_updateState) Task=`5542acd8-588c-41b7-b9f9-52e9826671ff`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:11:19,479::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:11:19,482::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:11:19,482::task::1191::Storage.TaskManager.Task::(prepare) Task=`5542acd8-588c-41b7-b9f9-52e9826671ff`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:11:19,482::task::595::Storage.TaskManager.Task::(_updateState) Task=`5542acd8-588c-41b7-b9f9-52e9826671ff`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:11:19,482::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:11:19,482::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:11:19,482::task::993::Storage.TaskManager.Task::(_decref) Task=`5542acd8-588c-41b7-b9f9-52e9826671ff`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:11:19,510::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:11:20,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:21,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091326 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:11:22,096::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:11:22,107::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688191 s, 539 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:23,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:23,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55915 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:11:25,203::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:11:25,215::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00130197 s, 402 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:25,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:26,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545306 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:11:27,021::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:11:27,022::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:11:28,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:28,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913668 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:30,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:30,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549895 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:11:31,584::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:11:32,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:11:32,275::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000638584 s, 581 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:32,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:33,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558834 s, 1.8 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:11:34,522::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:11:35,219::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:11:35,231::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646813 s, 809 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:35,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:35,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914509 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:11:37,032::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:11:37,033::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:11:37,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:38,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542817 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:40,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:40,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545665 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:11:42,278::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:11:42,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00067456 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:42,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:42,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912595 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:44,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:44,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913747 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:11:45,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:11:45,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070879 s, 738 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:46,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:11:47,043::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:11:47,044::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:11:47,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912894 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:49,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:49,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913153 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:11:49,534::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:11:51,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:51,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913491 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:11:52,293::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:11:52,305::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000719647 s, 516 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:53,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:53,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557399 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:11:55,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:11:55,322::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632232 s, 827 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:11:55,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:56,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912963 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:11:57,052::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:11:57,053::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:11:58,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:11:58,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556939 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:00,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:01,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543982 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:12:01,617::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:12:02,309::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:12:02,320::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000691492 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:03,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:03,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914066 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:12:04,546::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:12:05,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:12:05,411::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:05,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104312 s, 9.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:12:05,430::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643871 s, 812 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:12:07,064::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:12:07,065::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:12:07,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:07,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913154 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:09,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:09,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912913 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:11,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:11,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913535 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:12:12,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:12:12,335::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000666957 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:13,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:14,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557932 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:12:15,439::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:12:15,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720197 s, 726 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:16,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:16,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967119 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:12:17,075::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:12:17,076::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:12:18,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:18,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914007 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:12:19,549::task::595::Storage.TaskManager.Task::(_updateState) Task=`3d5f820e-392d-42a6-a824-238b2948d040`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:12:19,550::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:12:19,553::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:12:19,553::task::1191::Storage.TaskManager.Task::(prepare) Task=`3d5f820e-392d-42a6-a824-238b2948d040`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:12:19,553::task::595::Storage.TaskManager.Task::(_updateState) Task=`3d5f820e-392d-42a6-a824-238b2948d040`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:12:19,553::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:12:19,553::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:12:19,553::task::993::Storage.TaskManager.Task::(_decref) Task=`3d5f820e-392d-42a6-a824-238b2948d040`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:12:19,581::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:12:20,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:20,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091164 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:12:22,338::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:12:22,350::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661249 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:22,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:22,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091233 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:24,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:25,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913844 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:12:25,455::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:12:25,467::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000692825 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:27,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:12:27,088::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:12:27,089::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:12:27,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913234 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:29,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:29,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543658 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:12:31,649::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:12:31,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:31,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914038 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:12:32,354::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:12:32,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000634741 s, 584 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:33,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:33,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912245 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:12:34,593::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-29::DEBUG::2014-07-14 18:12:35,470::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:12:35,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000628756 s, 832 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:35,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:36,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926907 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:12:37,091::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:12:37,092::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:12:38,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:38,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912255 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:40,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:40,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913916 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:12:42,369::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:12:42,381::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667144 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:42,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:42,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09146 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:44,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:44,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914356 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:12:45,486::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:12:45,498::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000689775 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:46,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:46,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913142 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:12:47,103::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:12:47,104::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:12:48,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:48,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911923 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:12:49,606::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:12:50,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:51,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913887 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:12:52,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:12:52,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689017 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:53,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:53,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914133 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:55,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:12:55,518::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:12:56,066::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000693282 s, 754 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:12:56,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.553572 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:12:57,116::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:12:57,117::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:12:58,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:12:58,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.550843 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:00,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:00,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913851 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:13:01,681::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:13:02,399::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:13:02,410::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680082 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:02,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:02,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912938 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:13:04,617::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:13:04,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:05,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912599 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:13:06,074::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:13:06,086::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00136191 s, 384 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:07,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:07,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924292 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:13:07,132::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:13:07,136::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:13:09,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:09,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.50321 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:11,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:12,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544081 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:13:12,414::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:13:12,425::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000676672 s, 548 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:14,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:14,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559105 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:13:16,114::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:13:16,130::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00446912 s, 117 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:16,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:16,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911387 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:13:17,143::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:13:17,144::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:13:18,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:19,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913855 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:13:19,621::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc1ecde7-cde8-43fc-8619-00045c065e51`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:13:19,621::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:13:19,624::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:13:19,624::task::1191::Storage.TaskManager.Task::(prepare) Task=`fc1ecde7-cde8-43fc-8619-00045c065e51`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:13:19,624::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc1ecde7-cde8-43fc-8619-00045c065e51`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:13:19,625::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:13:19,625::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:13:19,625::task::993::Storage.TaskManager.Task::(_decref) Task=`fc1ecde7-cde8-43fc-8619-00045c065e51`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:13:19,664::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:13:21,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:21,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545096 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:13:22,429::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:13:22,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664717 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:23,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:23,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913055 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:25,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:13:26,137::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Dummy-49::DEBUG::2014-07-14 18:13:26,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54369 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:13:26,338::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:13:26,340::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 18:13:26,344::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 18:13:26,347::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 18:13:26,347::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 18:13:26,347::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 18:13:26,350::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:13:26,363::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000698242 s, 749 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:13:27,151::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:13:27,153::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:13:28,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:28,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912695 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:30,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:30,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091276 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:13:31,715::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:13:32,444::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:13:32,456::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00062637 s, 592 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:32,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:32,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912028 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:13:34,676::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:13:34,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:34,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913443 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:13:36,372::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:13:36,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000653177 s, 801 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:36,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:36,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912766 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:13:37,163::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:13:37,164::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:13:38,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:39,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912816 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:41,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:41,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543425 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:13:42,459::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:13:42,471::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066841 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:43,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:43,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912836 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:45,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:45,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0997669 s, 10.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:13:46,389::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:13:46,402::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068613 s, 762 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:13:47,173::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:13:47,174::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:13:48,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:48,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091412 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:13:49,690::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:13:50,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:50,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091204 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:52,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:52,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911093 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:13:52,473::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:13:52,489::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:13:52,491::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 18:13:52,494::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 18:13:52,496::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 18:13:52,496::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 18:13:52,496::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 18:13:52,497::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:13:52,508::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646485 s, 574 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:54,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:54,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091241 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:13:56,409::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:13:56,421::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065992 s, 793 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:13:56,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:57,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503423 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:13:57,185::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:13:57,187::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:13:59,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:13:59,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912115 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:01,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:01,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911738 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:14:01,747::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:14:02,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:14:02,524::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648771 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:03,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:03,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091613 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:14:04,702::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:14:05,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:05,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921434 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:14:06,425::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:14:06,437::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000723397 s, 723 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:14:07,196::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:14:07,197::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:14:07,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:07,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912532 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:09,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:10,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549462 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:12,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:12,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912522 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:14:12,527::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:14:12,539::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0006481 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:14,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:14,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913882 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:16,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:14:16,517::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:14:16,531::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00114339 s, 457 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:16,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.115803 s, 8.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:14:17,209::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:14:17,210::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:14:18,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:18,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912245 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:14:19,705::task::595::Storage.TaskManager.Task::(_updateState) Task=`a48f56cf-9ff3-4e75-8ceb-47e267a50867`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:14:19,706::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:14:19,708::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:14:19,708::task::1191::Storage.TaskManager.Task::(prepare) Task=`a48f56cf-9ff3-4e75-8ceb-47e267a50867`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:14:19,708::task::595::Storage.TaskManager.Task::(_updateState) Task=`a48f56cf-9ff3-4e75-8ceb-47e267a50867`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:14:19,709::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:14:19,709::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:14:19,709::task::993::Storage.TaskManager.Task::(_decref) Task=`a48f56cf-9ff3-4e75-8ceb-47e267a50867`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:14:19,736::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:14:20,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:20,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912496 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:14:22,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:14:22,553::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662428 s, 560 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:22,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:22,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912834 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:24,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:25,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091142 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:14:26,539::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:14:26,550::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679664 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:27,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:14:27,218::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:14:27,219::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:14:27,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543995 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:29,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:29,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912426 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:31,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 18:14:31,781::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:14:31,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912659 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:14:32,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:14:32,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670229 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:33,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:33,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912639 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:14:34,754::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:14:35,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:36,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542751 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:14:36,554::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:14:36,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640268 s, 817 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:14:37,226::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:14:37,228::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:14:38,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:39,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5326 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:41,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:41,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09322 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:14:42,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:14:42,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000718618 s, 516 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:43,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:43,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503037 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:45,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:46,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55011 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:14:46,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:14:46,581::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000679255 s, 770 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:14:47,239::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:14:47,240::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:14:48,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:48,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912722 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:14:49,767::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:14:50,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:50,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913391 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:14:52,588::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:14:52,599::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668266 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:52,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:52,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911579 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:54,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:54,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091317 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:14:56,585::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:14:56,596::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629541 s, 831 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:14:56,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:56,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912455 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:14:57,248::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:14:57,249::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:14:59,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:14:59,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913242 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:01,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:01,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544177 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:15:01,814::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:15:02,603::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:15:02,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679695 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:03,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:04,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.5453 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:15:04,779::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:15:06,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:06,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913971 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:15:06,600::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:15:06,612::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681337 s, 768 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:15:07,260::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:15:07,261::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:15:08,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:08,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912746 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:10,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:10,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913688 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:15:12,617::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:15:12,629::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000689493 s, 538 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:12,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:12,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091189 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:14,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:14,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10214 s, 10.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:15:16,616::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:15:16,627::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068992 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:16,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:17,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913325 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:15:17,266::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:15:17,267::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:15:19,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:19,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549803 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:15:19,782::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6032d68-2920-4944-842f-db883ceb41fa`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:15:19,782::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:15:19,785::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:15:19,785::task::1191::Storage.TaskManager.Task::(prepare) Task=`b6032d68-2920-4944-842f-db883ceb41fa`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:15:19,786::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6032d68-2920-4944-842f-db883ceb41fa`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:15:19,786::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:15:19,786::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:15:19,786::task::993::Storage.TaskManager.Task::(_decref) Task=`b6032d68-2920-4944-842f-db883ceb41fa`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:15:19,813::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:15:21,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:21,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912463 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:15:22,632::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:15:22,643::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690326 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:23,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:24,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.520489 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:26,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:15:26,873::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:15:26,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000751181 s, 696 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:26,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.561422 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:15:27,277::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:15:27,278::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:15:28,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:29,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.494959 s, 2.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:31,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:31,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911723 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:15:31,846::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:15:32,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:15:32,660::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637688 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:33,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:33,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913044 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:15:34,825::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:15:35,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:35,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921608 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:15:36,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:15:36,906::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000629619 s, 831 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:15:37,289::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:15:37,290::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:15:37,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:37,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913959 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:39,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:40,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091368 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:42,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:42,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915114 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:15:42,663::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:15:42,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667754 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:44,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:44,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912995 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:46,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:15:47,301::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:15:47,302::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:15:47,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:15:47,661::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000757764 s, 690 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:47,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.40343 s, 730 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:49,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:15:49,837::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:15:50,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540166 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:52,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:52,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0931797 s, 11.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:15:52,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:15:52,690::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000701881 s, 529 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:54,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:54,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911997 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:56,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:56,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912371 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:15:57,314::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:15:57,315::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:15:57,669::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:15:57,680::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657201 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:15:58,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:15:58,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911995 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:00,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:00,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912649 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:16:01,877::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:16:02,694::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:16:02,705::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000766564 s, 484 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:02,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:02,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913918 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:16:04,850::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:16:04,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:05,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914312 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:07,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:07,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912514 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:16:07,326::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:16:07,327::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:16:07,685::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:16:07,696::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00066485 s, 787 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:09,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:09,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912462 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:11,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:11,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913677 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:16:12,708::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:16:12,720::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682915 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:13,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:13,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913594 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:15,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:16,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.533796 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:16:17,336::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:16:17,337::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:16:17,700::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:16:17,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000708112 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:18,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:18,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913344 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:16:19,854::task::595::Storage.TaskManager.Task::(_updateState) Task=`df784e54-1757-4b24-9522-e0135552c31d`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:16:19,854::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:16:19,857::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:16:19,857::task::1191::Storage.TaskManager.Task::(prepare) Task=`df784e54-1757-4b24-9522-e0135552c31d`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:16:19,858::task::595::Storage.TaskManager.Task::(_updateState) Task=`df784e54-1757-4b24-9522-e0135552c31d`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:16:19,858::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:16:19,858::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:16:19,858::task::993::Storage.TaskManager.Task::(_decref) Task=`df784e54-1757-4b24-9522-e0135552c31d`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:16:19,883::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:16:20,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:20,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913399 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:22,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:22,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913347 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:16:22,723::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:16:22,735::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679748 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:24,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:24,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912528 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:26,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:26,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09119 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:16:27,348::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:16:27,349::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:16:27,716::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:16:27,727::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682609 s, 766 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:28,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:28,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912737 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:30,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:31,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912528 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:16:31,910::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:16:32,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:16:32,751::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000612746 s, 605 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:33,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:33,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.535761 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:16:34,894::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:16:35,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:36,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541658 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:16:37,360::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:16:37,361::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:16:37,731::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:16:37,745::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00226944 s, 230 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:38,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:38,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912691 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:40,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:40,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913873 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:42,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:42,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912244 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:16:42,754::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:16:42,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000642761 s, 577 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:44,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:44,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914233 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:46,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:46,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913781 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:16:47,372::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:16:47,373::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:16:47,748::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:16:47,760::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683465 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:48,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:49,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548536 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:16:49,906::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:16:51,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:51,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913714 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:16:52,770::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:16:52,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663649 s, 559 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:53,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:54,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541582 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:56,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:56,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556645 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:16:57,383::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:16:57,384::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:16:57,765::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:16:57,776::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000630054 s, 830 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:16:58,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:16:58,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913028 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:00,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:00,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920878 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:17:01,943::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:17:02,785::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:17:02,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000674521 s, 550 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:02,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:03,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915213 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:17:04,918::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:17:05,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:05,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911798 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:07,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:07,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912365 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:17:07,391::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:17:07,392::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:17:07,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:17:07,793::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00072547 s, 721 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:09,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:09,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912474 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:11,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:11,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913933 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:17:12,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:17:12,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000660988 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:13,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:13,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912127 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:15,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:15,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964653 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:17:17,403::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:17:17,404::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:17:17,797::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:17:17,809::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068358 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:17,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:17,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911716 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:17:19,921::task::595::Storage.TaskManager.Task::(_updateState) Task=`e0b2eb56-fb55-45bf-bbd0-00ab08379cc6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:17:19,922::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:17:19,926::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:17:19,926::task::1191::Storage.TaskManager.Task::(prepare) Task=`e0b2eb56-fb55-45bf-bbd0-00ab08379cc6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:17:19,926::task::595::Storage.TaskManager.Task::(_updateState) Task=`e0b2eb56-fb55-45bf-bbd0-00ab08379cc6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:17:19,926::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:17:19,926::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:17:19,927::task::993::Storage.TaskManager.Task::(_decref) Task=`e0b2eb56-fb55-45bf-bbd0-00ab08379cc6`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:17:19,952::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:17:20,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:20,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915747 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:22,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:22,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913904 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:17:22,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:17:22,826::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000666049 s, 557 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:24,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:24,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549195 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:26,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:26,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914171 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:17:27,415::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:17:27,416::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:17:27,812::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:17:27,824::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682025 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:28,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:29,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.517261 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:31,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 18:17:31,975::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:17:32,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55248 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:17:32,830::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:17:32,842::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000650037 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:34,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:34,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091355 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:17:34,966::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:17:36,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:36,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913171 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:17:37,423::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:17:37,424::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:17:37,828::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:17:37,840::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674882 s, 775 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:38,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:38,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912056 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:40,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:40,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915247 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:42,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:42,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912222 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:17:42,845::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:17:42,857::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621844 s, 597 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:44,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:44,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913708 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:46,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:46,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912324 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:17:47,435::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:17:47,436::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:17:47,844::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:17:47,856::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707482 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:48,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:49,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502827 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:17:49,979::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:17:51,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:51,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914251 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:17:52,860::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:17:52,871::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065154 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:53,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:53,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913561 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:55,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:56,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542557 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:17:57,448::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:17:57,449::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:17:57,867::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:17:57,879::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000978528 s, 534 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:17:58,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:17:58,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.549099 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:00,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:00,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911428 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:18:02,006::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:18:02,875::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:18:02,886::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643856 s, 576 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:02,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:03,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911387 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:18:04,992::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:18:05,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:05,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911877 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:07,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:18:07,460::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:18:07,462::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:18:07,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912363 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:18:07,883::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:18:07,894::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642001 s, 815 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:09,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:09,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913067 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:11,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:11,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913543 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:18:12,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:18:12,902::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000683478 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:13,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:13,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914017 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:15,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:16,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100404 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:18:17,473::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:18:17,474::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:18:17,898::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:18:17,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706239 s, 741 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:18,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:18,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912062 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:18:19,996::task::595::Storage.TaskManager.Task::(_updateState) Task=`f1acc4e1-5f2b-4bc0-8c25-d2248473a68c`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:18:19,996::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:18:19,999::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:18:19,999::task::1191::Storage.TaskManager.Task::(prepare) Task=`f1acc4e1-5f2b-4bc0-8c25-d2248473a68c`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:18:19,999::task::595::Storage.TaskManager.Task::(_updateState) Task=`f1acc4e1-5f2b-4bc0-8c25-d2248473a68c`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:18:19,999::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:18:19,999::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:18:20,000::task::993::Storage.TaskManager.Task::(_decref) Task=`f1acc4e1-5f2b-4bc0-8c25-d2248473a68c`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:18:20,027::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:18:20,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:20,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916282 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:22,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:22,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915846 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:18:22,905::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:18:22,916::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00064505 s, 575 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:24,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:24,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913692 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:26,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:26,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912017 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:18:27,485::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:18:27,486::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:18:27,913::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:18:27,928::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:18:27,929::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 18:18:27,932::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 18:18:27,933::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 18:18:27,934::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 18:18:27,934::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 18:18:27,935::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:18:27,946::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000685298 s, 763 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:28,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:28,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912635 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:30,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:31,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541919 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:18:32,039::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:18:32,920::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:18:32,933::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000624954 s, 594 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:33,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:33,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913059 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:18:35,039::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:18:35,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:35,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912275 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:18:37,497::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:18:37,498::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:18:37,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:37,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913031 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:18:37,950::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:18:37,962::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662328 s, 790 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:39,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:39,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914441 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:41,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:42,471::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.554675 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:18:42,936::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:18:42,947::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644019 s, 576 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:44,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:45,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541295 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:47,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:47,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913114 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:18:47,510::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:18:47,511::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:18:47,966::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:18:47,978::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000718011 s, 728 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:49,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:49,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913864 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:18:50,052::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:18:51,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:51,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917102 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:18:52,949::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:18:52,964::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:18:52,965::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 18:18:52,968::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 18:18:52,970::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 18:18:52,970::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 18:18:52,970::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 18:18:52,971::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:18:52,983::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678184 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:53,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:53,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915048 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:55,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:56,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507166 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:18:57,521::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:18:57,522::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:18:57,982::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:18:57,994::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00065601 s, 797 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:18:58,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:18:58,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915795 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:00,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:00,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915236 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:19:02,079::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:19:02,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:02,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541535 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:19:02,987::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:19:02,998::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644401 s, 576 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:04,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:19:05,064::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:19:05,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558907 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:07,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:19:07,533::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:19:07,535::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:19:07,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914048 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:19:07,998::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:19:08,010::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713289 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:09,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:10,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.502598 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:12,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:12,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915831 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:19:13,002::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:19:13,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637261 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:14,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:14,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913494 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:16,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:16,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548488 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:19:17,545::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:19:17,546::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:19:18,020::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:19:18,034::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680882 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:18,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:19,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091671 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:19:20,068::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b99df78-8df6-4755-8d4f-18bbf5499a4a`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:19:20,068::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:19:20,071::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:19:20,071::task::1191::Storage.TaskManager.Task::(prepare) Task=`3b99df78-8df6-4755-8d4f-18bbf5499a4a`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:19:20,071::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b99df78-8df6-4755-8d4f-18bbf5499a4a`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:19:20,071::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:19:20,071::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:19:20,071::task::993::Storage.TaskManager.Task::(_decref) Task=`3b99df78-8df6-4755-8d4f-18bbf5499a4a`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:19:20,097::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:19:21,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:21,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546182 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:19:23,017::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:19:23,028::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000641282 s, 579 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:23,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:23,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912517 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:25,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:26,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.515898 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:19:27,552::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:19:27,553::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:19:28,038::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:19:28,133::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000637032 s, 821 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:28,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:28,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913924 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:30,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:30,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912948 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:19:32,112::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:19:32,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:32,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914931 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:19:33,032::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:19:33,045::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00063631 s, 583 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:34,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:19:35,109::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:19:35,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519806 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:37,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:37,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913569 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:19:37,563::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:19:37,564::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:19:38,138::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:19:38,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664849 s, 787 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:39,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:39,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912015 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:41,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:41,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913663 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:19:43,048::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:19:43,061::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000630837 s, 588 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:43,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:44,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544517 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:46,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:46,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543817 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:19:47,575::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:19:47,576::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:19:48,154::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:19:48,165::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697454 s, 750 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:48,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:48,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916422 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:19:50,121::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:19:50,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:50,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914507 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:52,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:53,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09125 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:19:53,065::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:19:53,078::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000793214 s, 468 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:55,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:55,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912487 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:57,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:57,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912576 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:19:57,584::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:19:57,585::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:19:58,170::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:19:58,181::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676305 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:19:59,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:19:59,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914395 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:01,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:01,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913042 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:20:02,146::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:20:03,083::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:20:03,094::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000631198 s, 588 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:03,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:03,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912955 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:20:05,133::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:20:05,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:05,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914704 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:20:07,596::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:20:07,597::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:20:07,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:20:08,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:08,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54752 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:20:08,359::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00550577 s, 95.0 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:10,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:10,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091254 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:12,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:12,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912971 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:20:13,098::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:20:13,109::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000672882 s, 551 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:14,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:14,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918686 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:16,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:16,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912718 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:20:17,604::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:20:17,605::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:20:18,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:20:18,376::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062136 s, 842 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:18,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:19,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912427 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:20:20,136::task::595::Storage.TaskManager.Task::(_updateState) Task=`6456fe4b-ea1d-4db8-a305-8b002445ec2e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:20:20,136::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:20:20,139::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:20:20,139::task::1191::Storage.TaskManager.Task::(prepare) Task=`6456fe4b-ea1d-4db8-a305-8b002445ec2e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:20:20,140::task::595::Storage.TaskManager.Task::(_updateState) Task=`6456fe4b-ea1d-4db8-a305-8b002445ec2e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:20:20,140::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:20:20,140::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:20:20,140::task::993::Storage.TaskManager.Task::(_decref) Task=`6456fe4b-ea1d-4db8-a305-8b002445ec2e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:20:20,165::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:20:21,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:21,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912844 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:20:23,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:20:23,124::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000682699 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:23,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:23,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913024 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:25,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:25,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54815 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:20:27,616::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:20:27,617::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:20:27,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:27,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914231 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:20:28,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:20:28,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674759 s, 775 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:29,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:30,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913399 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:32,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 18:20:32,179::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:20:32,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542801 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:20:33,185::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:20:33,197::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000603429 s, 615 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:34,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:34,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914408 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:20:35,177::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:20:36,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:37,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.509523 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:20:37,624::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:20:37,625::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:20:38,397::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:20:38,409::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000632716 s, 827 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:39,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:39,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912923 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:41,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:41,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.547263 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:20:43,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:20:43,212::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000617032 s, 601 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:44,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:44,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913163 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:46,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:46,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0998578 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:20:47,631::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:20:47,632::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:20:48,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:20:48,481::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:48,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10347 s, 9.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:20:48,502::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676464 s, 773 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:20:50,188::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:20:50,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:50,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911301 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:52,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:52,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913521 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:20:53,216::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:20:53,227::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000679159 s, 546 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:54,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:54,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913693 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:56,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:56,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106355 s, 9.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:20:57,648::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:20:57,649::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:20:58,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:20:58,524::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669039 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:20:59,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:20:59,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913991 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:01,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:01,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913002 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:21:02,212::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:21:03,231::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:21:03,242::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000663111 s, 559 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:03,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:03,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091235 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:21:05,206::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:21:05,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:05,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913284 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:07,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:21:07,661::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:21:07,662::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:21:07,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911536 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:21:08,529::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:21:08,540::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000719619 s, 727 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:09,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:09,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091432 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:11,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:11,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911601 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:21:13,246::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:21:13,257::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000627688 s, 591 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:13,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:14,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913471 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:16,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:16,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101069 s, 10.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:21:17,668::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:21:17,669::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:21:18,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:18,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091062 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:21:18,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:21:18,555::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00073242 s, 714 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:21:20,209::task::595::Storage.TaskManager.Task::(_updateState) Task=`b5b3cf3f-ed95-467a-ab97-b18c6997861e`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:21:20,209::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:21:20,212::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:21:20,213::task::1191::Storage.TaskManager.Task::(prepare) Task=`b5b3cf3f-ed95-467a-ab97-b18c6997861e`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:21:20,213::task::595::Storage.TaskManager.Task::(_updateState) Task=`b5b3cf3f-ed95-467a-ab97-b18c6997861e`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:21:20,213::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:21:20,213::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:21:20,213::task::993::Storage.TaskManager.Task::(_decref) Task=`b5b3cf3f-ed95-467a-ab97-b18c6997861e`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:21:20,240::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:21:20,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:20,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913713 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:22,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:22,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091161 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:21:23,260::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:21:23,272::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000686922 s, 540 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:24,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:25,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548891 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:27,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:27,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913062 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:21:27,680::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:21:27,682::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:21:28,559::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:21:28,571::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000688383 s, 760 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:29,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:29,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911716 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:31,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:31,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091596 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:21:32,253::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:21:33,277::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:21:33,291::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621946 s, 597 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:33,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:33,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912269 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:21:35,252::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:21:35,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:35,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913983 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:21:37,688::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:21:37,689::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:21:37,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:37,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911772 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:21:38,575::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:21:38,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000801208 s, 653 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:39,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:40,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912936 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:42,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:42,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912005 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:21:43,298::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:21:43,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00064662 s, 574 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:44,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:44,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558185 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:46,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:46,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920963 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:21:47,696::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:21:47,697::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:21:48,591::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:21:48,603::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000678949 s, 770 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:48,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:48,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912575 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:21:50,264::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:21:50,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:51,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912737 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:53,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:53,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912897 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:21:53,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:21:53,325::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656335 s, 565 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:55,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:21:55,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912531 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:57,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:21:57,704::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:21:57,705::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:21:57,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543234 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:21:58,607::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:21:58,619::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00062939 s, 831 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:21:59,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:00,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.551055 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:22:02,284::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:22:02,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:02,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913935 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:22:03,328::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:22:03,340::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665003 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:04,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:04,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913899 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:22:05,276::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:22:06,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:06,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913169 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:22:07,712::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:22:07,713::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:22:08,622::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:22:08,635::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00165456 s, 316 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:08,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:08,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913295 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:10,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:11,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913378 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:13,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:13,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912612 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:22:13,343::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:22:13,354::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000678479 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:15,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:15,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912444 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:17,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:22:17,724::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:22:17,725::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:22:17,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541109 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:22:18,639::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:22:18,650::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720401 s, 726 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:19,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:19,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913329 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:22:20,279::task::595::Storage.TaskManager.Task::(_updateState) Task=`e46e1017-aeac-4a57-8d2d-7a3f5ec4c1ed`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:22:20,280::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:22:20,283::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:22:20,283::task::1191::Storage.TaskManager.Task::(prepare) Task=`e46e1017-aeac-4a57-8d2d-7a3f5ec4c1ed`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:22:20,283::task::595::Storage.TaskManager.Task::(_updateState) Task=`e46e1017-aeac-4a57-8d2d-7a3f5ec4c1ed`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:22:20,283::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:22:20,283::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:22:20,283::task::993::Storage.TaskManager.Task::(_decref) Task=`e46e1017-aeac-4a57-8d2d-7a3f5ec4c1ed`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:22:20,311::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:22:21,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:22:23,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:23,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.37566 s, 744 kB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:22:23,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000659204 s, 563 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:25,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:25,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912056 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:27,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:27,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912091 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:22:27,729::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:22:27,731::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:22:28,653::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:22:28,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00100704 s, 519 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:29,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:29,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912802 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:31,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:31,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091249 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:22:32,315::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:22:33,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:22:33,492::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000619708 s, 599 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:33,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:33,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914609 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:22:35,323::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:22:35,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:36,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914012 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:22:37,740::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:22:37,741::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:22:38,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:38,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914998 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:22:38,670::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:22:38,682::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673739 s, 776 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:40,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:40,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913648 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:42,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:42,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911614 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:22:43,495::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:22:43,507::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653895 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:44,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:44,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913436 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:46,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:47,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.500358 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:22:47,749::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:22:47,750::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:22:48,686::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:22:48,697::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000712378 s, 734 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:49,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:49,655::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543179 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:22:50,336::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:22:51,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:51,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091327 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:22:53,510::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:22:53,522::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000648653 s, 572 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:53,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:53,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091252 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:22:55,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:56,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917333 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:22:57,757::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:22:57,758::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:22:58,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:22:58,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55457 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:22:59,085::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:22:59,097::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000626403 s, 835 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:00,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:00,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912959 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:23:02,348::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:23:02,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:02,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915133 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:23:03,525::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:23:03,542::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00202979 s, 183 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:04,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:23:05,349::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:23:05,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544226 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:07,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:07,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927413 s, 11.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:23:07,767::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:23:07,768::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:23:09,102::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:23:09,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706414 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:09,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:09,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914601 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:11,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:11,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912107 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:23:13,546::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:23:13,557::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000698747 s, 531 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:13,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:13,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913648 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:15,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:15,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100686 s, 10.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:23:17,771::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:23:17,772::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:23:18,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:18,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912967 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:23:19,117::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:23:19,129::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.0006666 s, 785 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:20,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:23:20,352::task::595::Storage.TaskManager.Task::(_updateState) Task=`bebb3a6d-b081-4e28-bb8d-4a390c8059bf`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:23:20,353::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:23:20,786::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:23:20,787::task::1191::Storage.TaskManager.Task::(prepare) Task=`bebb3a6d-b081-4e28-bb8d-4a390c8059bf`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:23:20,787::task::595::Storage.TaskManager.Task::(_updateState) Task=`bebb3a6d-b081-4e28-bb8d-4a390c8059bf`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:23:20,788::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:23:20,789::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:23:20,790::task::993::Storage.TaskManager.Task::(_decref) Task=`bebb3a6d-b081-4e28-bb8d-4a390c8059bf`::ref 0 aborting False >Dummy-49::DEBUG::2014-07-14 18:23:20,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519125 s, 2.0 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:23:20,818::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:23:22,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:22,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921406 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:23:23,561::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:23:23,572::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000688826 s, 539 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:24,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:25,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911864 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:27,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:27,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912398 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:23:27,792::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:23:27,794::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:23:29,131::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:23:29,145::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:23:29,146::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 18:23:29,149::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 18:23:29,150::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 18:23:29,150::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 18:23:29,150::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 18:23:29,151::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:29,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:23:29,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000654946 s, 799 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:29,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0929695 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:31,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:31,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913183 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:23:32,381::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:23:33,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:33,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913251 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:23:33,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:23:33,589::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635378 s, 584 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:35,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:35,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091284 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:23:35,830::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:23:37,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:37,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914438 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:23:37,798::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:23:37,799::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:23:39,270::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:23:39,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000662948 s, 789 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:39,763::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:39,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915984 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:41,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:41,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913677 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:23:43,592::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:23:43,604::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000665315 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:43,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:44,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:46,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:46,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545876 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:23:47,810::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:23:47,811::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:23:48,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:48,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914377 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:23:49,286::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:23:49,297::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000725045 s, 721 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:50,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:23:50,843::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:23:50,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912662 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:52,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:53,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913138 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:23:53,606::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:23:53,619::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:23:53,620::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 18:23:53,623::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 18:23:53,625::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 18:23:53,625::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 18:23:53,625::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 18:23:53,626::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:23:53,638::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065087 s, 570 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:55,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:55,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.503585 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:57,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:57,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.207413 s, 4.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:23:57,822::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:23:57,823::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:23:59,302::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:23:59,314::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000642133 s, 814 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:23:59,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:23:59,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913717 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:01,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:02,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0933649 s, 11.0 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:24:02,412::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:24:03,641::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:24:03,653::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670507 s, 553 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:04,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:04,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091368 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:24:05,854::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:24:06,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:06,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091409 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:24:07,834::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:24:07,835::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:24:08,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:08,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913188 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:24:09,317::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:24:09,329::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000646482 s, 809 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:10,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:10,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913239 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:12,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:12,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914498 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:24:13,656::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:24:13,668::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670091 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:14,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:14,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091428 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:16,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:16,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911543 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:24:17,846::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:24:17,847::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:24:18,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:18,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912513 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:24:19,333::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:24:19,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000718425 s, 728 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:24:20,862::task::595::Storage.TaskManager.Task::(_updateState) Task=`83453d12-84a3-4678-90f5-a1df75c327e7`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:24:20,862::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:24:20,865::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:24:20,865::task::1191::Storage.TaskManager.Task::(prepare) Task=`83453d12-84a3-4678-90f5-a1df75c327e7`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:24:20,865::task::595::Storage.TaskManager.Task::(_updateState) Task=`83453d12-84a3-4678-90f5-a1df75c327e7`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:24:20,866::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:24:20,866::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:24:20,866::task::993::Storage.TaskManager.Task::(_decref) Task=`83453d12-84a3-4678-90f5-a1df75c327e7`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:24:20,895::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:24:20,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:21,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913113 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:23,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:23,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913406 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:24:23,671::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:24:23,683::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000649661 s, 571 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:25,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:25,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912812 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:27,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:27,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922433 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:24:27,858::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:24:27,859::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:24:29,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:24:29,360::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000703463 s, 743 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:29,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:29,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912665 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:31,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:31,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912454 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:24:32,444::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:24:33,686::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:33,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:24:33,707::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000635748 s, 584 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:34,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542448 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:24:35,907::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:24:36,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:36,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913647 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:24:37,871::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:24:37,872::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:24:38,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:38,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.530809 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:24:39,365::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:24:39,377::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00071475 s, 732 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:40,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:41,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927598 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:43,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:43,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548359 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:24:43,711::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:24:43,723::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00075365 s, 492 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:45,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:46,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545162 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:24:47,879::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:24:47,880::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:24:48,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:48,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913063 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:24:49,380::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:24:49,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000680536 s, 769 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:50,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:50,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912733 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:24:50,920::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:24:52,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:52,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911763 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:24:53,726::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:24:53,737::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667893 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:54,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:54,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09113 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:56,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:24:56,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913335 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:24:57,889::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:24:57,890::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:24:58,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:24:59,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:24:59,496::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635361 s, 823 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:24:59,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.569712 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:01,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:01,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912268 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:25:02,476::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:25:03,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:25:04,163::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:25:04,177::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00104758 s, 354 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:04,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.531559 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:25:05,935::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:25:06,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:06,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914366 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:25:07,901::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:25:07,902::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:25:08,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:08,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912492 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:25:09,507::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:25:09,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00071691 s, 730 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:10,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:10,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091363 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:12,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:12,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914051 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:25:14,182::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:25:14,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000697138 s, 532 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:14,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:14,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912418 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:16,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:16,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913365 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:25:17,913::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:25:17,914::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:25:18,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:19,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912995 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:25:19,523::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:25:19,534::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000664395 s, 787 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:25:20,938::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e65f0a6-9888-4f79-83b1-33100d18e4bd`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:25:20,939::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:25:20,942::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:25:20,942::task::1191::Storage.TaskManager.Task::(prepare) Task=`3e65f0a6-9888-4f79-83b1-33100d18e4bd`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:25:20,942::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e65f0a6-9888-4f79-83b1-33100d18e4bd`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:25:20,942::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:25:20,942::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:25:20,942::task::993::Storage.TaskManager.Task::(_decref) Task=`3e65f0a6-9888-4f79-83b1-33100d18e4bd`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:25:20,968::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:25:21,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:21,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914356 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:23,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:23,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912884 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:25:24,197::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:25:24,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00065967 s, 562 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:25,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:26,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559711 s, 1.8 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:25:27,925::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:25:27,926::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:25:28,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:28,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913415 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:25:29,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:25:29,549::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000717634 s, 729 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:30,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:30,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091178 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:32,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 18:25:32,511::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:25:32,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911621 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:25:34,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:25:34,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000647612 s, 573 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:34,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:34,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912226 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:25:35,980::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:25:36,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:36,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913083 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:25:37,933::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:25:37,934::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:25:38,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:39,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.505823 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:25:39,554::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:25:39,565::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000643309 s, 813 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:41,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:41,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914609 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:43,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:43,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.302972 s, 3.4 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:25:44,229::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:25:44,241::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656172 s, 565 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:45,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:45,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091371 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:25:47,945::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:25:47,947::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:25:48,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:48,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913758 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:25:49,569::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:25:49,581::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000677159 s, 772 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:50,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:50,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914411 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:25:50,992::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:25:52,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:52,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912861 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:25:54,244::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:25:54,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000637501 s, 582 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:54,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:54,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914269 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:25:56,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:56,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912848 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:25:57,958::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:25:57,959::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:25:58,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:25:59,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542129 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:25:59,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:25:59,598::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000614027 s, 852 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:01,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:01,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091563 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:26:02,542::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:26:03,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:03,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912517 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:26:04,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:26:04,271::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657851 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:05,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:05,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913863 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:26:06,003::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:26:07,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:07,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914136 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:26:07,970::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:26:07,971::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:26:09,601::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:26:09,613::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000691565 s, 756 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:09,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:09,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914072 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:11,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:11,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091816 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:13,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:13,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912759 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:26:14,274::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:26:14,285::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000670218 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:15,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:16,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0994098 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:26:17,977::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:26:17,978::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:26:18,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:18,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912772 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:26:19,617::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:26:19,628::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00068706 s, 761 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:20,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:20,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091117 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:26:21,006::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b05649b-fe53-44cd-90f3-364c5096e7c6`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:26:21,006::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:26:21,009::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:26:21,010::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b05649b-fe53-44cd-90f3-364c5096e7c6`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:26:21,010::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b05649b-fe53-44cd-90f3-364c5096e7c6`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:26:21,010::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:26:21,010::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:26:21,010::task::993::Storage.TaskManager.Task::(_decref) Task=`1b05649b-fe53-44cd-90f3-364c5096e7c6`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:26:21,039::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:26:22,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:22,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092055 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:26:24,288::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:26:24,300::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000677637 s, 547 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:24,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:24,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091221 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:26,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:26,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912918 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:26:27,989::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:26:27,990::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:26:28,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:28,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091379 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:26:29,632::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:26:29,644::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000676931 s, 773 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:30,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:31,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913653 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:26:32,574::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:26:33,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:33,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091569 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:26:34,303::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:26:34,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000664551 s, 558 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:35,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:35,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911991 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:26:36,052::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:26:37,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:37,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914361 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:26:37,997::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:26:37,998::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:26:39,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:39,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912514 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:26:39,647::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:26:39,659::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000655787 s, 798 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:41,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:41,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917156 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:43,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:44,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.55005 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:26:44,318::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:26:44,330::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000646253 s, 574 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:46,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:46,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0963566 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:26:48,009::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:26:48,010::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:26:48,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:48,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913114 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:26:49,663::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:26:49,675::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706636 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:50,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:50,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912239 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:26:51,064::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:26:52,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:53,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.557387 s, 1.8 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:26:54,334::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:26:54,345::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068315 s, 543 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:55,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:55,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911994 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:57,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:26:57,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914342 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:26:58,021::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:26:58,022::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:26:59,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:26:59,733::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:26:59,747::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682269 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:26:59,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108636 s, 9.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:01,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:01,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914499 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:27:02,607::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:27:03,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:03,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912843 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:27:04,349::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:27:04,361::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00066773 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:06,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:27:06,078::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:27:06,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091682 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:27:08,032::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:27:08,033::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:27:08,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:08,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913555 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:27:09,757::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:27:09,769::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000627435 s, 834 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:10,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:10,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091591 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:12,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:12,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915753 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:27:14,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:27:14,375::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000657625 s, 564 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:14,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:14,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.132632 s, 7.7 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:16,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:17,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543554 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:27:18,045::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:27:18,046::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:27:19,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:19,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919254 s, 11.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:27:19,773::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:27:19,784::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000697141 s, 750 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:27:21,082::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f7368cb-96f1-45a0-8635-00c762f32ca8`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:27:21,082::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:27:21,085::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:27:21,085::task::1191::Storage.TaskManager.Task::(prepare) Task=`0f7368cb-96f1-45a0-8635-00c762f32ca8`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:27:21,086::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f7368cb-96f1-45a0-8635-00c762f32ca8`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:27:21,086::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:27:21,086::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:27:21,086::task::993::Storage.TaskManager.Task::(_decref) Task=`0f7368cb-96f1-45a0-8635-00c762f32ca8`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:27:21,113::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:27:21,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:21,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914209 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:23,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:23,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913078 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:27:24,379::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:27:24,390::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000654945 s, 566 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:25,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:26,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.548408 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:27:28,057::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:27:28,058::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:27:28,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:28,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913863 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:27:29,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:27:29,800::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000709108 s, 738 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:30,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:30,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091492 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:32,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:32,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913779 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:27:32,643::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:27:34,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:27:34,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626798 s, 592 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:34,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:34,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913573 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:27:36,125::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:27:36,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:36,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0923132 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:27:38,065::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:27:38,066::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:27:38,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:38,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0969573 s, 10.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:27:39,804::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:27:39,816::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000660437 s, 792 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:40,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:40,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916045 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:42,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:43,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925952 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:27:44,410::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:27:44,421::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621775 s, 597 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:45,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:45,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913614 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:47,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:47,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913981 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:27:48,079::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:27:48,080::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:27:49,323::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:49,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914069 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:27:49,820::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:27:49,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674398 s, 776 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:27:51,138::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:27:51,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:51,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913402 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:53,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:53,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911976 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:27:54,425::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:27:54,436::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000611423 s, 607 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:27:55,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:56,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.500418 s, 2.0 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:27:58,086::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:27:58,087::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:27:58,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:27:58,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0932582 s, 11.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:27:59,835::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:27:59,847::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000674399 s, 776 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:00,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:00,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912855 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:02,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:02,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913748 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:28:02,675::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Thread-77::DEBUG::2014-07-14 18:28:04,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:28:04,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00068143 s, 544 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:04,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:04,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916388 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:28:06,150::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:28:06,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:06,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09124 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:28:08,099::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:28:08,100::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:28:08,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:08,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091233 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:28:09,851::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:28:09,863::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000706546 s, 740 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:11,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:11,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.511951 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:13,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:13,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091389 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:28:14,454::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:28:14,465::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000643481 s, 577 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:15,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:15,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0981773 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:17,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:18,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0989872 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:28:18,110::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:28:18,111::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::DEBUG::2014-07-14 18:28:19,866::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:28:19,878::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000709256 s, 737 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:20,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:20,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913436 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:28:21,154::task::595::Storage.TaskManager.Task::(_updateState) Task=`1e6b406f-bf69-413b-a191-9b308302d5bd`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:28:21,154::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:28:21,157::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:28:21,157::task::1191::Storage.TaskManager.Task::(prepare) Task=`1e6b406f-bf69-413b-a191-9b308302d5bd`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:28:21,157::task::595::Storage.TaskManager.Task::(_updateState) Task=`1e6b406f-bf69-413b-a191-9b308302d5bd`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:28:21,157::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:28:21,158::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:28:21,158::task::993::Storage.TaskManager.Task::(_decref) Task=`1e6b406f-bf69-413b-a191-9b308302d5bd`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:28:21,184::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:28:22,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:22,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913515 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:24,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:28:24,869::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:28:24,882::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000690314 s, 537 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:24,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.553097 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:26,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:27,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912995 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:28:28,124::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:28:28,125::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:28:29,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:29,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914572 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:28:29,880::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:28:29,893::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:28:29,895::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 18:28:29,898::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 18:28:29,899::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 18:28:29,899::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 18:28:29,899::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 18:28:29,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:28:29,912::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000681231 s, 768 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:31,241::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:31,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914265 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:28:32,708::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:28:33,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:33,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912685 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:28:34,887::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:28:34,901::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644067 s, 576 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:35,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:35,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913333 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:28:36,196::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:28:37,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:37,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913778 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:28:38,136::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:28:38,137::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:28:39,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-29::DEBUG::2014-07-14 18:28:40,226::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:28:40,240::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000640633 s, 816 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:40,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.524697 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:42,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:42,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.543896 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:44,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:28:45,023::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:28:45,038::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000810252 s, 458 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:45,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.197623 s, 5.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:47,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:28:48,144::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:28:48,145::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:28:48,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.3886 s, 737 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:28:50,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:28:50,259::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000695893 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:50,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:51,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540519 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:28:51,210::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:28:53,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:53,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091539 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:28:55,045::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:28:55,059::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:28:55,060::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 18:28:55,063::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Thread-77::DEBUG::2014-07-14 18:28:55,065::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 18:28:55,065::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 18:28:55,065::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 18:28:55,066::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:28:55,078::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000666804 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:55,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:55,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914285 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:28:57,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:57,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913904 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:28:58,155::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:28:58,156::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:28:59,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:28:59,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913568 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:29:00,263::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:29:00,275::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673367 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:01,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:01,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913858 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:29:02,745::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:29:03,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:03,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913448 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:29:05,082::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:29:05,093::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669496 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:05,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:05,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912968 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:29:06,222::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:29:07,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:29:08,168::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:29:08,169::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:29:08,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.519092 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:29:10,279::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:29:10,290::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000705598 s, 741 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:10,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:10,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913795 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:12,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:12,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914149 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:14,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:29:15,162::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:29:15,175::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680711 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:15,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.525699 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:17,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:17,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091651 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:29:18,176::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:29:18,178::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:29:19,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:19,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913712 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:29:20,294::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:29:20,306::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000728597 s, 718 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:29:21,225::task::595::Storage.TaskManager.Task::(_updateState) Task=`122b9cd3-3bcc-4abe-b11a-08f0b1c7eab8`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:29:21,226::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:29:21,229::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:29:21,229::task::1191::Storage.TaskManager.Task::(prepare) Task=`122b9cd3-3bcc-4abe-b11a-08f0b1c7eab8`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:29:21,229::task::595::Storage.TaskManager.Task::(_updateState) Task=`122b9cd3-3bcc-4abe-b11a-08f0b1c7eab8`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:29:21,229::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:29:21,229::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:29:21,230::task::993::Storage.TaskManager.Task::(_decref) Task=`122b9cd3-3bcc-4abe-b11a-08f0b1c7eab8`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:29:21,255::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:29:21,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:21,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913411 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:23,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:23,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912407 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:29:25,179::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:29:25,191::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661012 s, 561 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:25,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:25,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919584 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:27,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:27,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09145 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:29:28,189::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:29:28,190::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:29:29,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:30,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0928424 s, 11.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:29:30,310::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:29:30,321::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000682176 s, 767 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:32,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:32,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558484 s, 1.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:29:32,784::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:29:34,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:34,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914495 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:29:35,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:29:35,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000621068 s, 597 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:29:36,266::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:29:36,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:36,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914994 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:29:38,201::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:29:38,202::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:29:38,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:38,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913407 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:29:40,325::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:29:40,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650503 s, 804 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:41,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:41,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916631 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:43,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:43,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914863 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:29:45,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:29:45,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000612825 s, 605 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:45,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:45,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.507673 s, 2.0 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:47,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:48,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925878 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:29:48,213::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:29:48,214::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:29:50,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:50,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914196 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:29:50,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:29:50,353::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000696103 s, 751 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:29:51,278::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:29:52,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:52,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918357 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:54,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:54,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912615 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:29:55,225::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:29:55,237::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000668519 s, 555 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:29:56,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:56,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918954 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:29:58,225::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:29:58,227::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:29:58,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:29:58,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912552 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:30:00,357::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:30:00,369::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000656768 s, 796 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:00,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:00,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915119 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:02,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >VM Channels Listener::DEBUG::2014-07-14 18:30:02,817::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:30:02,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912565 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:04,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:04,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912952 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:30:05,240::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:30:05,253::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00150748 s, 246 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:30:06,290::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:30:06,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:07,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540114 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:30:08,238::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:30:08,239::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:30:09,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:10,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541122 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:30:10,373::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:30:10,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000720558 s, 726 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:12,119::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:12,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915191 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:14,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:14,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913596 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:30:15,256::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:30:15,268::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000712774 s, 521 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:16,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:16,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0998893 s, 10.3 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:30:18,248::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:30:18,249::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:30:18,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:18,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914069 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:30:20,389::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:30:20,400::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000710861 s, 736 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:20,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:20,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914166 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:30:21,293::task::595::Storage.TaskManager.Task::(_updateState) Task=`0df1be82-5744-44a9-8b86-4ebd2b2d74af`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:30:21,294::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:30:21,297::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:30:21,297::task::1191::Storage.TaskManager.Task::(prepare) Task=`0df1be82-5744-44a9-8b86-4ebd2b2d74af`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:30:21,297::task::595::Storage.TaskManager.Task::(_updateState) Task=`0df1be82-5744-44a9-8b86-4ebd2b2d74af`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:30:21,297::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:30:21,297::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:30:21,298::task::993::Storage.TaskManager.Task::(_decref) Task=`0df1be82-5744-44a9-8b86-4ebd2b2d74af`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:30:21,325::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:30:22,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:22,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091383 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:24,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:25,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913809 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:30:25,271::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:30:25,282::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000644548 s, 576 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:27,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:27,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914888 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:30:28,261::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:30:28,262::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:30:29,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:29,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.568167 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:30:30,404::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:30:30,415::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00070053 s, 747 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:31,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:31,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911709 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:30:32,847::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:30:33,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:34,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912871 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:30:35,286::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:30:35,298::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000628512 s, 590 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:36,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:36,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914397 s, 11.2 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:30:36,338::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:30:38,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:30:38,273::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:30:38,275::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:30:38,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.504293 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:30:40,419::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:30:40,431::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694747 s, 753 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:40,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:40,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915624 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:42,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:42,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913235 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:44,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:45,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913683 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:30:45,301::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:30:45,313::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000626019 s, 593 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:47,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:47,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912072 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:30:48,286::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:30:48,287::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:30:49,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:49,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912314 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:30:50,435::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:30:50,446::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069521 s, 752 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:51,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:30:51,362::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:30:51,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915343 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:53,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:53,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912711 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:30:55,317::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:30:55,328::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000700954 s, 529 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:55,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:30:55,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913086 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:30:57,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >GuestMonitor-vm11::ERROR::2014-07-14 18:30:58,298::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:30:58,299::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:30:59,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36989 s, 748 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:31:00,451::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:31:00,463::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000657791 s, 795 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:01,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:01,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091316 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:31:02,880::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:31:03,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:03,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913629 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:05,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:31:05,337::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:31:05,885::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00074254 s, 500 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:05,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.552317 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:31:06,375::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:31:07,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:08,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912143 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:31:08,311::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:31:08,312::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:31:10,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:10,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914564 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:31:10,468::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:31:10,480::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000669132 s, 782 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:12,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:12,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912337 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:14,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:14,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913761 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:31:15,890::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:31:15,904::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000669953 s, 554 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:16,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:16,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0965717 s, 10.6 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:31:18,322::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:31:18,323::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:31:18,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:18,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914814 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:31:20,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:31:20,495::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000690993 s, 757 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:20,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:21,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.544221 s, 1.9 MB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:31:21,378::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a33ea3e-1fc6-4b1d-b1b9-e78c77a6a0e2`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:31:21,378::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:31:21,381::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:31:21,381::task::1191::Storage.TaskManager.Task::(prepare) Task=`5a33ea3e-1fc6-4b1d-b1b9-e78c77a6a0e2`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:31:21,381::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a33ea3e-1fc6-4b1d-b1b9-e78c77a6a0e2`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:31:21,382::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:31:21,382::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:31:21,382::task::993::Storage.TaskManager.Task::(_decref) Task=`5a33ea3e-1fc6-4b1d-b1b9-e78c77a6a0e2`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:31:21,409::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:31:23,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:23,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912902 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:25,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:25,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911723 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:31:25,910::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:31:25,921::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000680286 s, 545 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:27,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:27,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913999 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:31:28,332::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:31:28,333::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:31:29,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:29,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912522 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:31:30,498::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:31:30,510::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000713189 s, 733 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:31,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:32,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540484 s, 1.9 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:31:32,928::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:31:34,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:34,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.546455 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:31:35,925::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:31:35,937::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000656317 s, 565 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:31:36,422::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:31:36,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:37,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.541308 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:31:38,339::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:31:38,340::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:31:39,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:39,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914531 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:31:40,514::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:31:40,526::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000649318 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:41,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:41,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091442 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:43,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:43,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091459 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:45,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:45,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0967754 s, 10.6 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:31:45,941::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:31:45,956::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000606936 s, 611 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:48,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:48,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913734 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:31:48,345::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:31:48,346::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:31:50,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:50,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913469 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:31:50,529::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:31:50,541::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000686603 s, 762 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:31:51,435::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:31:52,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:52,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925377 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:54,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:55,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542839 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:31:55,964::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:31:55,975::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000652187 s, 569 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:31:57,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:57,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091283 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:31:58,357::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:31:58,358::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:31:59,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:31:59,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913099 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:32:00,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:32:00,556::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000619456 s, 844 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:01,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:02,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.36273 s, 751 kB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:32:02,970::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:32:04,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:05,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913078 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:32:05,979::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:32:05,991::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000694151 s, 534 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:32:06,447::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:32:07,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:07,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915018 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:32:08,366::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:32:08,367::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:32:09,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:09,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914172 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:32:10,561::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:32:10,573::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000673493 s, 777 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:11,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:11,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091437 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:13,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:13,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912805 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:15,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:15,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914411 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:32:15,996::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:32:16,013::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00489307 s, 75.8 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:17,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:17,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920214 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:32:18,379::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:32:18,380::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:32:19,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:19,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914296 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:32:20,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:32:20,588::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000687021 s, 761 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:32:21,451::task::595::Storage.TaskManager.Task::(_updateState) Task=`19ef6601-a75b-4b3c-8762-7eb7d34f780f`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:32:21,451::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:32:21,454::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:32:21,454::task::1191::Storage.TaskManager.Task::(prepare) Task=`19ef6601-a75b-4b3c-8762-7eb7d34f780f`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:32:21,454::task::595::Storage.TaskManager.Task::(_updateState) Task=`19ef6601-a75b-4b3c-8762-7eb7d34f780f`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:32:21,454::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:32:21,454::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:32:21,455::task::993::Storage.TaskManager.Task::(_decref) Task=`19ef6601-a75b-4b3c-8762-7eb7d34f780f`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:32:21,482::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:32:21,988::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:23,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.35117 s, 758 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:25,369::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:25,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910553 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:32:26,018::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:32:26,030::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653868 s, 567 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:27,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:27,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913337 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:32:28,387::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:32:28,389::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:32:29,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:30,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.54391 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:32:30,592::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:32:30,603::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000668244 s, 783 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:32,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:32,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912019 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:32:33,010::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:32:34,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:34,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09143 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:32:36,034::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:32:36,047::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00064757 s, 573 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:36,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-113::DEBUG::2014-07-14 18:32:36,503::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:32:36,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092103 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:32:38,399::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:32:38,400::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:32:38,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:38,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913294 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:32:40,608::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:32:40,620::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000683706 s, 765 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:40,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:41,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.542599 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:43,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:43,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911951 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:45,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:45,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916812 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:32:46,052::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:32:46,070::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00459188 s, 80.8 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:47,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:47,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913808 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:32:48,412::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:32:48,413::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:32:49,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:49,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912754 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:32:50,624::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:32:50,645::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000749298 s, 698 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:32:51,516::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:32:51,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:51,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916067 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:53,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:53,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922728 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:55,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:56,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911579 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:32:56,206::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:32:56,218::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000705273 s, 526 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:32:58,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:32:58,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914087 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:32:58,424::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:32:58,425::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:33:00,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:00,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913098 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:33:00,653::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:33:00,665::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000622431 s, 840 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:02,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:02,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919758 s, 11.1 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:33:03,042::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:33:04,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:04,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.540282 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:33:06,222::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:33:06,233::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000661441 s, 561 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:33:06,529::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:33:06,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:07,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912499 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:33:08,436::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:33:08,437::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:33:09,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:09,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091238 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:33:10,669::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:33:10,681::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000707944 s, 739 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:11,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:11,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091285 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:13,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:13,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912419 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:15,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:15,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915154 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:33:16,238::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:33:16,255::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00445103 s, 83.4 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:17,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:17,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912499 s, 11.2 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:33:18,445::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:33:18,446::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:33:19,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:19,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912397 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:33:20,685::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:33:20,696::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000694979 s, 753 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:33:21,532::task::595::Storage.TaskManager.Task::(_updateState) Task=`e912a88e-5e3c-41b4-abbf-e8a1f9044471`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:33:21,532::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::INFO::2014-07-14 18:33:21,535::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:33:21,536::task::1191::Storage.TaskManager.Task::(prepare) Task=`e912a88e-5e3c-41b4-abbf-e8a1f9044471`::finished: {'truesize': '512', 'apparentsize': '1073741824'} >Thread-113::DEBUG::2014-07-14 18:33:21,536::task::595::Storage.TaskManager.Task::(_updateState) Task=`e912a88e-5e3c-41b4-abbf-e8a1f9044471`::moving from state preparing -> state finished >Thread-113::DEBUG::2014-07-14 18:33:21,536::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:33:21,536::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:33:21,536::task::993::Storage.TaskManager.Task::(_decref) Task=`e912a88e-5e3c-41b4-abbf-e8a1f9044471`::ref 0 aborting False >Thread-113::DEBUG::2014-07-14 18:33:21,563::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:33:21,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:21,879::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911988 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:23,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:23,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915155 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:26,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:33:26,538::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:26,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.528168 s, 1.9 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:33:26,553::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00222633 s, 167 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:33:28,457::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:33:28,458::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:33:28,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:29,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.505073 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:33:30,699::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:33:30,712::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:33:30,713::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-07-14 18:33:30,716::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=f248ab94-c2d2-4d14-b120-b2fa7cc96393:Active,1f0449ec-49a1-4c13-b973-a24382d1db5b:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data4', 'ROLE=Master', 'SDUUID=1f0449ec-49a1-4c13-b973-a24382d1db5b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ac94889c0976650b3dc12fd58741cf690f92da8'] >Thread-29::DEBUG::2014-07-14 18:33:30,718::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::INFO::2014-07-14 18:33:30,718::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_imageNS already registered >Thread-29::INFO::2014-07-14 18:33:30,718::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 1f0449ec-49a1-4c13-b973-a24382d1db5b_volumeNS already registered >Thread-29::DEBUG::2014-07-14 18:33:30,719::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:33:30,730::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.00069284 s, 755 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:31,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:31,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913577 s, 11.2 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:33:33,075::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:33:33,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:33,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.559621 s, 1.8 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:35,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:36,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.517095 s, 2.0 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:33:36,559::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:33:36,572::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000662467 s, 560 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:33:36,581::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:33:38,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:38,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920156 s, 11.1 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:33:38,470::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:33:38,471::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:33:40,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:40,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915529 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:33:40,734::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:33:40,746::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000650007 s, 805 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:42,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:42,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919515 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:44,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:44,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914519 s, 11.2 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:33:46,575::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:33:46,587::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000667499 s, 556 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:46,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:47,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545702 s, 1.9 MB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:33:48,482::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:33:48,483::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:33:49,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:49,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.506933 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:33:50,750::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:33:50,761::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000612746 s, 854 kB/s\n'; <rc> = 0 >Thread-113::DEBUG::2014-07-14 18:33:51,594::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Dummy-49::DEBUG::2014-07-14 18:33:51,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:52,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091831 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:54,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:54,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914241 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:33:56,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-77::DEBUG::2014-07-14 18:33:56,589::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:33:56,617::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:33:56,698::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-07-14 18:33:56,701::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data5', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data5', 'ROLE=Regular', 'SDUUID=f248ab94-c2d2-4d14-b120-b2fa7cc96393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=11531b7b97e3f900baafe8c705d4c576f3aa91c4'] >Dummy-49::DEBUG::2014-07-14 18:33:56,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0924348 s, 11.1 MB/s\n'; <rc> = 0 >Thread-77::DEBUG::2014-07-14 18:33:56,706::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-07-14 18:33:56,707::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_imageNS already registered >Thread-77::INFO::2014-07-14 18:33:56,708::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f248ab94-c2d2-4d14-b120-b2fa7cc96393_volumeNS already registered >Thread-77::DEBUG::2014-07-14 18:33:56,712::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-77::DEBUG::2014-07-14 18:33:56,727::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00279693 s, 133 kB/s\n'; <rc> = 0 >GuestMonitor-vm11::ERROR::2014-07-14 18:33:58,494::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:33:58,495::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Dummy-49::DEBUG::2014-07-14 18:33:58,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:33:58,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913043 s, 11.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-07-14 18:34:00,766::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-07-14 18:34:00,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n523 bytes (523 B) copied, 0.000635788 s, 823 kB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:34:00,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:34:00,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0910885 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-49::DEBUG::2014-07-14 18:34:02,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:34:03,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.116625 s, 8.8 MB/s\n'; <rc> = 0 >VM Channels Listener::DEBUG::2014-07-14 18:34:03,107::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 24. >Dummy-49::DEBUG::2014-07-14 18:34:05,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-49::DEBUG::2014-07-14 18:34:05,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) FAILED: <err> = "dd: failed to open '/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox': No such file or directory\n"; <rc> = 1 >Dummy-49::INFO::2014-07-14 18:34:05,794::storage_mailbox::794::Storage.MailBox.SpmMailMonitor::(run) SPM_MailMonitor - Incoming mail monitoring thread stopped >Thread-113::DEBUG::2014-07-14 18:34:06,607::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::ERROR::2014-07-14 18:34:06,734::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/fileSD.py", line 596, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393' >Thread-77::DEBUG::2014-07-14 18:34:06,735::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 changed its status to Invalid >Thread-11503::DEBUG::2014-07-14 18:34:06,735::misc::816::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-11503::DEBUG::2014-07-14 18:34:06,736::misc::826::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-11503::DEBUG::2014-07-14 18:34:06,736::misc::826::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_upgradePoolDomain` >Thread-11503::DEBUG::2014-07-14 18:34:06,737::misc::836::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >GuestMonitor-vm11::ERROR::2014-07-14 18:34:08,505::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:34:08,506::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::ERROR::2014-07-14 18:34:10,790::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1f0449ec-49a1-4c13-b973-a24382d1db5b monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/fileSD.py", line 596, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b' >Thread-29::DEBUG::2014-07-14 18:34:10,790::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 1f0449ec-49a1-4c13-b973-a24382d1db5b changed its status to Invalid >Thread-11506::DEBUG::2014-07-14 18:34:10,791::misc::816::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-11506::DEBUG::2014-07-14 18:34:10,791::misc::826::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-11506::DEBUG::2014-07-14 18:34:10,791::misc::826::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_upgradePoolDomain` >Thread-11506::DEBUG::2014-07-14 18:34:10,792::misc::836::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-77::ERROR::2014-07-14 18:34:16,737::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/fileSD.py", line 596, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393' >GuestMonitor-vm11::ERROR::2014-07-14 18:34:18,508::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >GuestMonitor-vm11::ERROR::2014-07-14 18:34:18,509::vm::2860::vm.Vm::(_getRunningVmStats) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Error fetching vm stats >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2854, in _getRunningVmStats > decStats = self._vmStats.get() > File "/usr/share/vdsm/virt/vm.py", line 604, in get > self._getUserCpuTuneInfo(stats) > File "/usr/share/vdsm/virt/vm.py", line 453, in _getUserCpuTuneInfo > if eInfo['vcpuLimit']: >KeyError: 'vcpuLimit' >Thread-29::ERROR::2014-07-14 18:34:20,793::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1f0449ec-49a1-4c13-b973-a24382d1db5b monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/fileSD.py", line 596, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b' >Thread-113::DEBUG::2014-07-14 18:34:21,611::task::595::Storage.TaskManager.Task::(_updateState) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::moving from state init -> state preparing >Thread-113::INFO::2014-07-14 18:34:21,611::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-113::ERROR::2014-07-14 18:34:21,612::task::866::Storage.TaskManager.Task::(_setError) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 3059, in getVolumeSize > apparentsize = str(dom.getVSize(imgUUID, volUUID)) > File "/usr/share/vdsm/storage/fileSD.py", line 314, in getVSize > return self.oop.os.stat(volPath).st_size > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73' >Thread-113::DEBUG::2014-07-14 18:34:21,613::task::885::Storage.TaskManager.Task::(_run) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::Task._run: 9e062c8a-db18-4d3d-95fe-0550d78308e7 ('1f0449ec-49a1-4c13-b973-a24382d1db5b', '2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'da775677-8785-45a4-a0de-0ad497e0adb6', 'eb353c32-6147-4ab3-b8a7-85b5603ada73') {} failed - stopping task >Thread-113::DEBUG::2014-07-14 18:34:21,613::task::1217::Storage.TaskManager.Task::(stop) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::stopping in state preparing (force False) >Thread-113::DEBUG::2014-07-14 18:34:21,613::task::993::Storage.TaskManager.Task::(_decref) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::ref 1 aborting True >Thread-113::INFO::2014-07-14 18:34:21,613::task::1171::Storage.TaskManager.Task::(prepare) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::aborting: Task is aborted: u"[Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73'" - code 100 >Thread-113::DEBUG::2014-07-14 18:34:21,614::task::1176::Storage.TaskManager.Task::(prepare) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::Prepare: aborted: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73' >Thread-113::DEBUG::2014-07-14 18:34:21,614::task::993::Storage.TaskManager.Task::(_decref) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::ref 0 aborting True >Thread-113::DEBUG::2014-07-14 18:34:21,614::task::928::Storage.TaskManager.Task::(_doAbort) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::Task._doAbort: force False >Thread-113::DEBUG::2014-07-14 18:34:21,614::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::DEBUG::2014-07-14 18:34:21,614::task::595::Storage.TaskManager.Task::(_updateState) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::moving from state preparing -> state aborting >Thread-113::DEBUG::2014-07-14 18:34:21,614::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::_aborting: recover policy none >Thread-113::DEBUG::2014-07-14 18:34:21,615::task::595::Storage.TaskManager.Task::(_updateState) Task=`9e062c8a-db18-4d3d-95fe-0550d78308e7`::moving from state aborting -> state failed >Thread-113::DEBUG::2014-07-14 18:34:21,615::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-113::DEBUG::2014-07-14 18:34:21,615::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-113::ERROR::2014-07-14 18:34:21,615::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73' >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73' >Thread-113::ERROR::2014-07-14 18:34:21,615::vm::4060::vm.Vm::(updateDriveVolume) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Unable to update the volume eb353c32-6147-4ab3-b8a7-85b5603ada73 (domain: 1f0449ec-49a1-4c13-b973-a24382d1db5b image: da775677-8785-45a4-a0de-0ad497e0adb6) for the drive vda >Thread-113::DEBUG::2014-07-14 18:34:21,650::libvirtconnection::125::root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata element is not present >Thread-77::ERROR::2014-07-14 18:34:26,738::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/fileSD.py", line 596, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 199, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5/f248ab94-c2d2-4d14-b120-b2fa7cc96393' >MainThread::DEBUG::2014-07-14 18:34:27,044::vdsm::56::vds::(sigtermHandler) Received signal 15 >MainThread::DEBUG::2014-07-14 18:34:27,045::protocoldetector::133::vds.MultiProtocolAcceptor::(stop) Stopping Acceptor >MainThread::INFO::2014-07-14 18:34:27,045::__init__::556::jsonrpc.JsonRpcServer::(stop) Stopping JsonRPC Server >Detector thread::DEBUG::2014-07-14 18:34:27,046::protocoldetector::104::vds.MultiProtocolAcceptor::(_cleanup) Cleaning Acceptor >JsonRpcServer::DEBUG::2014-07-14 18:34:27,046::__init__::497::jsonrpc.JsonRpcServer::(serve_requests) Popped request >MainThread::INFO::2014-07-14 18:34:27,047::vmchannels::188::vds::(stop) VM channels listener was stopped. >MainThread::INFO::2014-07-14 18:34:27,048::momIF::91::MOM::(stop) Shutting down MOM >MainThread::DEBUG::2014-07-14 18:34:27,048::task::595::Storage.TaskManager.Task::(_updateState) Task=`5af93fa1-1d96-41bf-afe7-5fcdaa9cf13c`::moving from state init -> state preparing >MainThread::INFO::2014-07-14 18:34:27,048::logUtils::44::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) >Thread-11::DEBUG::2014-07-14 18:34:27,049::storageServer::706::Storage.ConnectionMonitor::(_monitorConnections) Monitoring stopped >VM Channels Listener::INFO::2014-07-14 18:34:27,133::vmchannels::183::vds::(run) VM channels listener thread has ended. >Thread-40::INFO::2014-07-14 18:34:28,485::storage_mailbox::509::Storage.MailBox.HsmMailMonitor::(run) HSM_MailboxMonitor - Incoming mail monitoring thread stopped, clearing outgoing mail >Thread-40::INFO::2014-07-14 18:34:28,485::storage_mailbox::385::Storage.MailBox.HsmMailMonitor::(_sendMail) HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=512', 'seek=8'] >Thread-40::DEBUG::2014-07-14 18:34:28,485::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) '/usr/bin/dd of=/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox iflag=fullblock oflag=direct conv=notrunc bs=512 seek=8' (cwd None) >Thread-40::DEBUG::2014-07-14 18:34:28,497::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED: <err> = "/usr/bin/dd: failed to open '/rhev/data-center/2d31dd81-c30f-45b1-8933-2aeeae6ca427/mastersd/dom_md/inbox': No such file or directory\n"; <rc> = 1 >MainThread::INFO::2014-07-14 18:34:29,796::domainMonitor::125::Storage.DomainMonitor::(close) Stopping domain monitors >MainThread::INFO::2014-07-14 18:34:29,796::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::DEBUG::2014-07-14 18:34:29,796::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for f248ab94-c2d2-4d14-b120-b2fa7cc96393 >Thread-77::INFO::2014-07-14 18:34:29,797::clusterlock::206::Storage.SANLock::(releaseHostId) Releasing host id for domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 (id: 1) >Thread-29::DEBUG::2014-07-14 18:34:30,794::remoteFileHandler::260::Storage.RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-29::ERROR::2014-07-14 18:34:30,794::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1f0449ec-49a1-4c13-b973-a24382d1db5b monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/fileSD.py", line 596, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > self._sendAll(pack(LENGTH_STRUCT_FMT, len(request)), timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 173, in _sendAll > timeLeft): > File "/usr/lib64/python2.7/contextlib.py", line 84, in helper > return GeneratorContextManager(func(*args, **kwds)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 140, in _poll > raise Timeout() >Timeout >Thread-29::DEBUG::2014-07-14 18:34:30,795::remoteFileHandler::260::Storage.RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-77::DEBUG::2014-07-14 18:34:31,797::clusterlock::216::Storage.SANLock::(releaseHostId) Host id for domain f248ab94-c2d2-4d14-b120-b2fa7cc96393 released successfully (id: 1) >MainThread::INFO::2014-07-14 18:34:31,798::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::DEBUG::2014-07-14 18:34:31,798::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-29::INFO::2014-07-14 18:34:31,798::clusterlock::206::Storage.SANLock::(releaseHostId) Releasing host id for domain 1f0449ec-49a1-4c13-b973-a24382d1db5b (id: 1) >Thread-29::DEBUG::2014-07-14 18:34:31,798::domainMonitor::184::Storage.DomainMonitorThread::(_monitorLoop) Unable to release the host id 1 for domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 181, in _monitorLoop > self.domain.releaseHostId(self.hostId, unused=True) > File "/usr/share/vdsm/storage/sd.py", line 470, in releaseHostId > self._clusterLock.releaseHostId(hostId, async, unused) > File "/usr/share/vdsm/storage/clusterlock.py", line 213, in releaseHostId > raise se.ReleaseHostIdFailure(self._sdUUID, e) >ReleaseHostIdFailure: Cannot release host id: ('1f0449ec-49a1-4c13-b973-a24382d1db5b', SanlockException(16, 'Sanlock lockspace remove failure', 'Device or resource busy')) >MainThread::DEBUG::2014-07-14 18:34:31,799::taskManager::90::Storage.TaskManager::(prepareForShutdown) Request to stop all tasks >MainThread::INFO::2014-07-14 18:34:31,799::logUtils::47::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None >MainThread::DEBUG::2014-07-14 18:34:31,799::task::1191::Storage.TaskManager.Task::(prepare) Task=`5af93fa1-1d96-41bf-afe7-5fcdaa9cf13c`::finished: None >MainThread::DEBUG::2014-07-14 18:34:31,800::task::595::Storage.TaskManager.Task::(_updateState) Task=`5af93fa1-1d96-41bf-afe7-5fcdaa9cf13c`::moving from state preparing -> state finished >MainThread::DEBUG::2014-07-14 18:34:31,800::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2014-07-14 18:34:31,800::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2014-07-14 18:34:31,800::task::993::Storage.TaskManager.Task::(_decref) Task=`5af93fa1-1d96-41bf-afe7-5fcdaa9cf13c`::ref 0 aborting False >MainThread::INFO::2014-07-14 18:34:31,800::vdsm::135::vds::(run) VDSM main thread ended. Waiting for 12 other threads... >MainThread::INFO::2014-07-14 18:34:31,800::vdsm::139::vds::(run) <Thread(libvirtEventLoop, started daemon 140300997146368)> >MainThread::INFO::2014-07-14 18:34:31,800::vdsm::139::vds::(run) <WorkerThread(Thread-10, started daemon 140300576077568)> >MainThread::INFO::2014-07-14 18:34:31,800::vdsm::139::vds::(run) <WorkerThread(Thread-9, started daemon 140300584470272)> >MainThread::INFO::2014-07-14 18:34:31,801::vdsm::139::vds::(run) <WorkerThread(Thread-8, started daemon 140300592862976)> >MainThread::INFO::2014-07-14 18:34:31,801::vdsm::139::vds::(run) <WorkerThread(Thread-7, started daemon 140300601255680)> >MainThread::INFO::2014-07-14 18:34:31,801::vdsm::139::vds::(run) <WorkerThread(Thread-6, started daemon 140300609648384)> >MainThread::INFO::2014-07-14 18:34:31,801::vdsm::139::vds::(run) <_MainThread(MainThread, started 140301099312960)> >MainThread::INFO::2014-07-14 18:34:31,801::vdsm::139::vds::(run) <WorkerThread(Thread-5, started daemon 140300955182848)> >MainThread::INFO::2014-07-14 18:34:31,801::vdsm::139::vds::(run) <WorkerThread(Thread-4, started daemon 140300963575552)> >MainThread::INFO::2014-07-14 18:34:31,801::vdsm::139::vds::(run) <WorkerThread(Thread-3, started daemon 140300971968256)> >MainThread::DEBUG::2014-07-14 18:34:31,801::sampling::368::vm.Vm::(stop) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Stop statistics collection >Thread-113::DEBUG::2014-07-14 18:34:31,802::sampling::399::vm.Vm::(run) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Stats thread finished >MainThread::INFO::2014-07-14 18:34:31,802::vdsm::139::vds::(run) <VmStatsThread(Thread-113, started daemon 140299680380672)> >MainThread::INFO::2014-07-14 18:34:31,802::vdsm::139::vds::(run) <WorkerThread(87ae615b-b044-4bfb-9344-a6b2668f9df1, started daemon 140300988753664)> >MainThread::INFO::2014-07-14 18:34:31,802::vdsm::139::vds::(run) <WorkerThread(b16a8011-d0a1-4334-9df9-677d433cd737, started daemon 140300980360960)> >MainThread::INFO::2014-07-14 18:49:45,190::vdsm::129::vds::(run) (PID: 18833) I am the actual vdsm 4.15.0-136.git02f4fd5.fc19 dhcp-4-198.tlv.redhat.com (3.11.10-301.fc20.x86_64) >MainThread::DEBUG::2014-07-14 18:49:45,191::resourceManager::421::Storage.ResourceManager::(registerNamespace) Registering namespace 'Storage' >MainThread::DEBUG::2014-07-14 18:49:45,192::threadPool::35::Storage.ThreadPool::(__init__) Enter - numThreads: 10, waitTimeout: 3, maxTasks: 500 >MainThread::DEBUG::2014-07-14 18:49:45,195::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt >MainThread::WARNING::2014-07-14 18:49:45,195::fileUtils::149::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists >MainThread::DEBUG::2014-07-14 18:49:45,229::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm >MainThread::DEBUG::2014-07-14 18:49:45,237::multipath::141::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving >MainThread::DEBUG::2014-07-14 18:49:45,237::hsm::416::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /usr/sbin/lvm dumpconfig global/locking_type' (cwd None) >MainThread::DEBUG::2014-07-14 18:49:45,263::hsm::416::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2014-07-14 18:49:45,264::hsm::440::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' >MainThread::DEBUG::2014-07-14 18:49:45,270::hsm::472::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] >MainThread::DEBUG::2014-07-14 18:49:45,271::hsm::473::Storage.HSM::(__cleanStorageRepository) Mount list: ['/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_sla_kobi_data14', '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4', '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data5'] >MainThread::DEBUG::2014-07-14 18:49:45,271::hsm::475::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers >MainThread::DEBUG::2014-07-14 18:49:45,273::hsm::518::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' >storageRefresh::DEBUG::2014-07-14 18:49:45,273::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >MainThread::INFO::2014-07-14 18:49:45,275::dispatcher::46::Storage.Dispatcher::(__init__) Starting StorageDispatcher... >storageRefresh::DEBUG::2014-07-14 18:49:45,275::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-07-14 18:49:45,293::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >storageRefresh::DEBUG::2014-07-14 18:49:45,293::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-07-14 18:49:45,294::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >storageRefresh::DEBUG::2014-07-14 18:49:45,294::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R' (cwd None) >storageRefresh::DEBUG::2014-07-14 18:49:45,341::misc::746::Storage.SamplingMethod::(__call__) Returning last result >storageRefresh::DEBUG::2014-07-14 18:49:45,346::multipath::109::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /usr/sbin/multipath' (cwd None) >MainThread::DEBUG::2014-07-14 18:49:45,422::task::595::Storage.TaskManager.Task::(_updateState) Task=`a52ad8d9-563c-4544-8e7f-8b94957ed767`::moving from state init -> state preparing >MainThread::INFO::2014-07-14 18:49:45,422::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x11dd4d0>>) >storageRefresh::DEBUG::2014-07-14 18:49:45,423::multipath::109::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >MainThread::INFO::2014-07-14 18:49:45,423::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None >storageRefresh::DEBUG::2014-07-14 18:49:45,423::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >MainThread::DEBUG::2014-07-14 18:49:45,424::task::1191::Storage.TaskManager.Task::(prepare) Task=`a52ad8d9-563c-4544-8e7f-8b94957ed767`::finished: None >storageRefresh::DEBUG::2014-07-14 18:49:45,424::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >MainThread::DEBUG::2014-07-14 18:49:45,424::task::595::Storage.TaskManager.Task::(_updateState) Task=`a52ad8d9-563c-4544-8e7f-8b94957ed767`::moving from state preparing -> state finished >storageRefresh::DEBUG::2014-07-14 18:49:45,424::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >MainThread::DEBUG::2014-07-14 18:49:45,425::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >storageRefresh::DEBUG::2014-07-14 18:49:45,425::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >MainThread::DEBUG::2014-07-14 18:49:45,425::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >storageRefresh::DEBUG::2014-07-14 18:49:45,425::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >MainThread::DEBUG::2014-07-14 18:49:45,426::task::993::Storage.TaskManager.Task::(_decref) Task=`a52ad8d9-563c-4544-8e7f-8b94957ed767`::ref 0 aborting False >storageRefresh::DEBUG::2014-07-14 18:49:45,426::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-07-14 18:49:45,427::misc::746::Storage.SamplingMethod::(__call__) Returning last result >storageRefresh::DEBUG::2014-07-14 18:49:45,427::lvm::317::Storage.OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-07-14 18:49:45,430::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) >MainThread::INFO::2014-07-14 18:49:45,438::momIF::49::MOM::(__init__) Starting up MOM >MainThread::INFO::2014-07-14 18:49:45,442::vmchannels::192::vds::(settimeout) Setting channels' timeout to 30 seconds. >VM Channels Listener::INFO::2014-07-14 18:49:45,455::vmchannels::175::vds::(run) Starting VM channels listener thread. >clientIFinit::DEBUG::2014-07-14 18:49:45,462::libvirtconnection::146::root::(get) trying to connect libvirt >MainThread::DEBUG::2014-07-14 18:49:45,473::protocoldetector::129::vds.MultiProtocolAcceptor::(add_detector) adding detector: <rpc.BindingXMLRPC.XmlDetector instance at 0x143b8c0> >storageRefresh::DEBUG::2014-07-14 18:49:45,480::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0 >storageRefresh::DEBUG::2014-07-14 18:49:45,480::lvm::342::Storage.OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-07-14 18:49:45,481::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-07-14 18:49:45,481::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-07-14 18:49:45,528::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-07-14 18:49:45,528::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-07-14 18:49:45,535::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None) >storageRefresh::DEBUG::2014-07-14 18:49:45,566::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-07-14 18:49:45,568::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-07-14 18:49:45,572::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-07-14 18:49:45,612::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-07-14 18:49:45,614::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-07-14 18:49:45,618::hsm::385::Storage.HSM::(storageRefresh) HSM is ready >clientIFinit::DEBUG::2014-07-14 18:49:45,631::clientIF::547::vds::(_recoverVm) Trying to recover dc518130-d5f0-455f-b5eb-76f471f41cf4 >clientIFinit::INFO::2014-07-14 18:49:45,632::clientIF::393::vds::(createVm) vmContainerLock acquired by vm dc518130-d5f0-455f-b5eb-76f471f41cf4 >MainThread::DEBUG::2014-07-14 18:49:45,635::protocoldetector::129::vds.MultiProtocolAcceptor::(add_detector) adding detector: <yajsonrpc.stompReactor.StompDetector instance at 0x1ba8950> >JsonRpcServer::DEBUG::2014-07-14 18:49:45,636::__init__::495::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Detector thread::DEBUG::2014-07-14 18:49:45,640::protocoldetector::69::vds.MultiProtocolAcceptor::(serve_forever) Acceptor running >clientIFinit::DEBUG::2014-07-14 18:49:45,641::libvirtconnection::146::root::(get) trying to connect libvirt >Detector thread::DEBUG::2014-07-14 18:49:45,642::protocoldetector::71::vds.MultiProtocolAcceptor::(serve_forever) Using required_size=7 >Thread-13::DEBUG::2014-07-14 18:49:45,671::vm::2220::vm.Vm::(_startUnderlyingVm) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Start >clientIFinit::DEBUG::2014-07-14 18:49:45,671::clientIF::406::vds::(createVm) Total desktops after creation of dc518130-d5f0-455f-b5eb-76f471f41cf4 is 1 >Thread-13::DEBUG::2014-07-14 18:49:45,674::vm::2224::vm.Vm::(_startUnderlyingVm) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::_ongoingCreations acquired >Thread-13::INFO::2014-07-14 18:49:45,681::vm::3220::vm.Vm::(_run) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::VM wrapper has started >Thread-13::WARNING::2014-07-14 18:49:45,690::vm::2012::vm.Vm::(buildConfDevices) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}' found >Thread-13::WARNING::2014-07-14 18:49:45,698::vm::2012::vm.Vm::(buildConfDevices) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Unknown type found, device: '{'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}' found >Thread-13::WARNING::2014-07-14 18:49:45,745::vm::2012::vm.Vm::(buildConfDevices) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Unknown type found, device: '{'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}' found >Thread-13::DEBUG::2014-07-14 18:49:45,750::task::595::Storage.TaskManager.Task::(_updateState) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::moving from state init -> state preparing >Thread-13::INFO::2014-07-14 18:49:45,761::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID='eb353c32-6147-4ab3-b8a7-85b5603ada73', options=None) >Thread-13::ERROR::2014-07-14 18:49:45,771::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-13::ERROR::2014-07-14 18:49:45,787::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-13::DEBUG::2014-07-14 18:49:45,793::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-13::DEBUG::2014-07-14 18:49:45,831::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1f0449ec-49a1-4c13-b973-a24382d1db5b' (cwd None) >Thread-13::DEBUG::2014-07-14 18:49:45,860::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n Volume group "1f0449ec-49a1-4c13-b973-a24382d1db5b" not found\n Skipping volume group 1f0449ec-49a1-4c13-b973-a24382d1db5b\n'; <rc> = 5 >Thread-13::WARNING::2014-07-14 18:49:45,861::lvm::370::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', ' Volume group "1f0449ec-49a1-4c13-b973-a24382d1db5b" not found', ' Skipping volume group 1f0449ec-49a1-4c13-b973-a24382d1db5b'] >Thread-13::DEBUG::2014-07-14 18:49:45,861::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-13::ERROR::2014-07-14 18:49:46,021::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1f0449ec-49a1-4c13-b973-a24382d1db5b not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('1f0449ec-49a1-4c13-b973-a24382d1db5b',) >Thread-13::ERROR::2014-07-14 18:49:46,022::task::866::Storage.TaskManager.Task::(_setError) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 3058, in getVolumeSize > dom = sdCache.produce(sdUUID=sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('1f0449ec-49a1-4c13-b973-a24382d1db5b',) >Thread-13::DEBUG::2014-07-14 18:49:46,024::task::885::Storage.TaskManager.Task::(_run) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::Task._run: 64182a02-0f8f-4f29-ac64-81db8f91ffcd ('1f0449ec-49a1-4c13-b973-a24382d1db5b', '2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'da775677-8785-45a4-a0de-0ad497e0adb6', 'eb353c32-6147-4ab3-b8a7-85b5603ada73') {} failed - stopping task >Thread-13::DEBUG::2014-07-14 18:49:46,024::task::1217::Storage.TaskManager.Task::(stop) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::stopping in state preparing (force False) >Thread-13::DEBUG::2014-07-14 18:49:46,024::task::993::Storage.TaskManager.Task::(_decref) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::ref 1 aborting True >Thread-13::INFO::2014-07-14 18:49:46,024::task::1171::Storage.TaskManager.Task::(prepare) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 >Thread-13::DEBUG::2014-07-14 18:49:46,024::task::1176::Storage.TaskManager.Task::(prepare) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::Prepare: aborted: Storage domain does not exist >Thread-13::DEBUG::2014-07-14 18:49:46,024::task::993::Storage.TaskManager.Task::(_decref) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::ref 0 aborting True >Thread-13::DEBUG::2014-07-14 18:49:46,025::task::928::Storage.TaskManager.Task::(_doAbort) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::Task._doAbort: force False >Thread-13::DEBUG::2014-07-14 18:49:46,025::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-07-14 18:49:46,025::task::595::Storage.TaskManager.Task::(_updateState) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::moving from state preparing -> state aborting >Thread-13::DEBUG::2014-07-14 18:49:46,025::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::_aborting: recover policy none >Thread-13::DEBUG::2014-07-14 18:49:46,025::task::595::Storage.TaskManager.Task::(_updateState) Task=`64182a02-0f8f-4f29-ac64-81db8f91ffcd`::moving from state aborting -> state failed >Thread-13::DEBUG::2014-07-14 18:49:46,025::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-07-14 18:49:46,025::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::ERROR::2014-07-14 18:49:46,025::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: ('1f0449ec-49a1-4c13-b973-a24382d1db5b',)", 'code': 358}} >Thread-13::DEBUG::2014-07-14 18:49:46,055::vm::5216::vm.Vm::(_getUnderlyingDriveInfo) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Looking for drive with attributes {'name': u'hdc', 'bootOrder': '', 'boot': [], 'readonly': True, 'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'path': '', 'type': u'cdrom'} >Thread-13::DEBUG::2014-07-14 18:49:46,056::vm::5237::vm.Vm::(_getUnderlyingDriveInfo) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Matched {'name': (u'hdc', u'hdc'), 'bootOrder': ('', None), 'boot': ([], None), 'readonly': (True, True), 'address': ({u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}), 'path': ('', ''), 'type': (u'cdrom', u'cdrom')} >Thread-13::DEBUG::2014-07-14 18:49:46,056::vm::5256::vm.Vm::(_getUnderlyingDriveInfo) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Matched {'name': (u'hdc', None), 'bootOrder': ('', None), 'boot': ([], None), 'readonly': (True, None), 'address': ({u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, None), 'path': ('', None), 'type': (u'cdrom', None)} >Thread-13::DEBUG::2014-07-14 18:49:46,056::vm::5216::vm.Vm::(_getUnderlyingDriveInfo) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Looking for drive with attributes {'name': u'vda', 'bootOrder': '', 'boot': [], 'readonly': False, 'address': {u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'path': u'/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73', 'type': u'disk'} >Thread-13::DEBUG::2014-07-14 18:49:46,057::vm::5237::vm.Vm::(_getUnderlyingDriveInfo) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Matched {'name': (u'vda', u'vda'), 'bootOrder': ('', None), 'boot': ([], None), 'readonly': (False, False), 'address': ({u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, {u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}), 'path': (u'/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73', '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73'), 'type': (u'disk', u'disk')} >Thread-13::DEBUG::2014-07-14 18:49:46,057::vm::5256::vm.Vm::(_getUnderlyingDriveInfo) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Matched {'name': (u'vda', None), 'bootOrder': ('', None), 'boot': ([], None), 'readonly': (False, None), 'address': ({u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, None), 'path': (u'/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73', None), 'type': (u'disk', None)} >Thread-13::DEBUG::2014-07-14 18:49:46,141::vm::1350::vm.Vm::(blockDev) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Unable to determine if the path '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73' is a block device >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 1347, in blockDev > self._blockDev = utils.isBlockDevice(self.path) > File "/usr/lib64/python2.7/site-packages/vdsm/utils.py", line 104, in isBlockDevice > return stat.S_ISBLK(os.stat(path).st_mode) >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data4/1f0449ec-49a1-4c13-b973-a24382d1db5b/images/da775677-8785-45a4-a0de-0ad497e0adb6/eb353c32-6147-4ab3-b8a7-85b5603ada73' >Thread-13::INFO::2014-07-14 18:49:46,143::vm::2243::vm.Vm::(_startUnderlyingVm) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Skipping errors on recovery >Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2227, in _startUnderlyingVm > self._run() > File "/usr/share/vdsm/virt/vm.py", line 3294, in _run > self._domDependentInit() > File "/usr/share/vdsm/virt/vm.py", line 3185, in _domDependentInit > self._syncVolumeChain(drive) > File "/usr/share/vdsm/virt/vm.py", line 5686, in _syncVolumeChain > volumes = self._driveGetActualVolumeChain(drive) > File "/usr/share/vdsm/virt/vm.py", line 5665, in _driveGetActualVolumeChain > sourceAttr = ('file', 'dev')[drive.blockDev] >TypeError: tuple indices must be integers, not NoneType >Thread-13::DEBUG::2014-07-14 18:49:46,143::vm::2246::vm.Vm::(_startUnderlyingVm) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::_ongoingCreations released >clientIFinit::DEBUG::2014-07-14 18:49:46,677::task::595::Storage.TaskManager.Task::(_updateState) Task=`ebfb8f2b-aee7-402e-86fe-b1aaca6a1195`::moving from state init -> state preparing >clientIFinit::INFO::2014-07-14 18:49:46,677::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >clientIFinit::INFO::2014-07-14 18:49:46,678::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:49:46,678::task::1191::Storage.TaskManager.Task::(prepare) Task=`ebfb8f2b-aee7-402e-86fe-b1aaca6a1195`::finished: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:49:46,678::task::595::Storage.TaskManager.Task::(_updateState) Task=`ebfb8f2b-aee7-402e-86fe-b1aaca6a1195`::moving from state preparing -> state finished >clientIFinit::DEBUG::2014-07-14 18:49:46,678::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >clientIFinit::DEBUG::2014-07-14 18:49:46,678::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >clientIFinit::DEBUG::2014-07-14 18:49:46,678::task::993::Storage.TaskManager.Task::(_decref) Task=`ebfb8f2b-aee7-402e-86fe-b1aaca6a1195`::ref 0 aborting False >clientIFinit::DEBUG::2014-07-14 18:49:51,684::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f7c3fb3-f23d-49a1-a828-f32009a01a72`::moving from state init -> state preparing >clientIFinit::INFO::2014-07-14 18:49:51,684::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >clientIFinit::INFO::2014-07-14 18:49:51,685::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:49:51,685::task::1191::Storage.TaskManager.Task::(prepare) Task=`9f7c3fb3-f23d-49a1-a828-f32009a01a72`::finished: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:49:51,685::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f7c3fb3-f23d-49a1-a828-f32009a01a72`::moving from state preparing -> state finished >clientIFinit::DEBUG::2014-07-14 18:49:51,685::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >clientIFinit::DEBUG::2014-07-14 18:49:51,685::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >clientIFinit::DEBUG::2014-07-14 18:49:51,685::task::993::Storage.TaskManager.Task::(_decref) Task=`9f7c3fb3-f23d-49a1-a828-f32009a01a72`::ref 0 aborting False >Detector thread::DEBUG::2014-07-14 18:49:51,692::protocoldetector::159::vds.MultiProtocolAcceptor::(_add_connection) Adding connection from 10.35.4.70:54487 >Detector thread::DEBUG::2014-07-14 18:49:51,693::protocoldetector::170::vds.MultiProtocolAcceptor::(_remove_connection) Connection removed from 10.35.4.70:54487 >Detector thread::DEBUG::2014-07-14 18:49:51,693::protocoldetector::196::vds.MultiProtocolAcceptor::(_handle_connection_read) Detected protocol xml from 10.35.4.70:54487 >Detector thread::DEBUG::2014-07-14 18:49:51,694::BindingXMLRPC::1147::XmlDetector::(handleSocket) xml over http detected from ('10.35.4.70', 54487) >Thread-19::DEBUG::2014-07-14 18:49:51,695::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call getCapabilities with () {} >Thread-19::DEBUG::2014-07-14 18:49:51,770::utils::717::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,781::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:51,806::utils::717::root::(execCmd) '/usr/bin/sudo -n /usr/bin/tee /sys/class/net/bonding_masters' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,824::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:51,825::utils::717::root::(execCmd) '/usr/bin/sudo -n /usr/bin/tee /sys/class/net/lPQHgJBNWAHlZXv/bonding/mode' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,848::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:51,851::utils::717::root::(execCmd) '/usr/bin/sudo -n /usr/bin/tee /sys/class/net/lPQHgJBNWAHlZXv/bonding/mode' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,867::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:51,869::utils::717::root::(execCmd) '/usr/bin/sudo -n /usr/bin/tee /sys/class/net/lPQHgJBNWAHlZXv/bonding/mode' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,884::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:51,887::utils::717::root::(execCmd) '/usr/bin/sudo -n /usr/bin/tee /sys/class/net/lPQHgJBNWAHlZXv/bonding/mode' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,904::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:51,906::utils::717::root::(execCmd) '/usr/bin/sudo -n /usr/bin/tee /sys/class/net/lPQHgJBNWAHlZXv/bonding/mode' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,921::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:51,924::utils::717::root::(execCmd) '/usr/bin/sudo -n /usr/bin/tee /sys/class/net/lPQHgJBNWAHlZXv/bonding/mode' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,940::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:51,942::utils::717::root::(execCmd) '/usr/bin/sudo -n /usr/bin/tee /sys/class/net/lPQHgJBNWAHlZXv/bonding/mode' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,958::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:51,960::utils::717::root::(execCmd) '/usr/bin/sudo -n /usr/bin/tee /sys/class/net/bonding_masters' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:51,988::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:52,020::utils::717::root::(execCmd) 'numactl --hardware' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:52,030::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:52,031::utils::717::root::(execCmd) 'sysctl -n -e kernel.numa_balancing' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:52,041::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:52,051::BindingXMLRPC::1114::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:d2dedd7a362'}], 'FC': []}, 'packages2': {'kernel': {'release': '301.fc20.x86_64', 'buildtime': 1386244877.0, 'version': '3.11.10'}, 'spice-server': {'release': '3.fc20', 'buildtime': 1383129969L, 'version': '0.12.4'}, 'vdsm': {'release': '136.git02f4fd5.fc19', 'buildtime': 1403104797L, 'version': '4.15.0'}, 'qemu-kvm': {'release': '4.fc20', 'buildtime': 1398798279L, 'version': '1.6.2'}, 'libvirt': {'release': '4.fc20', 'buildtime': 1395169169L, 'version': '1.1.3.4'}, 'qemu-img': {'release': '4.fc20', 'buildtime': 1398798279L, 'version': '1.6.2'}, 'mom': {'release': '1.master.fc19', 'buildtime': 1403103991L, 'version': '0.4.0'}}, 'numaNodeDistance': {'0': [10]}, 'cpuModel': 'Intel(R) Core(TM)2 Quad CPU Q8400 @ 2.66GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'selinux': {'mode': '1'}, 'kdumpStatus': 0, 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '10.35.4.198', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'gateway': '10.35.5.254', 'bootproto4': 'dhcp', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv4addrs': ['10.35.4.198/23'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.4.198', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv4addrs': ['10.35.4.198/23'], 'ipv6gateway': '::', 'gateway': '10.35.5.254', 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '18', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.180373ae5fe3', 'bridge_id': '8000.180373ae5fe3', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '863', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'multicast_query_interval': '12500', 'multicast_query_use_ifaddr': '0', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}, 'ports': ['em1']}}, 'uuid': '44454C4C-3800-1056-8032-C8C04F34354A', 'lastClientIface': 'ovirtmgmt', 'nics': {'p2p2': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'p2p2', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '7ab5c17e-4d3d-4d9b-86ad-31e8e31eb4be', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': '00:07:E9:A5:1A:AF', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:07:e9:a5:1a:af', 'speed': 0}, 'em1': {'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '18:03:73:ae:5f:e3', 'DEVICE': 'em1', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '18:03:73:ae:5f:e3', 'speed': 100}, 'p2p1': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'p2p1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '77c39be3-75f9-438a-b26a-ca3253e979fb', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': '00:07:E9:A5:1A:AE', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:07:e9:a5:1a:ae', 'speed': 0}}, 'software_revision': '136', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,xsave,lahf_lm,dtherm,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:d2dedd7a362', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'autoNumaBalancing': 2, 'reservedMem': '321', 'bondings': {'bond0': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '5a:15:34:d5:9a:79', 'slaves': [], 'options': {}}}, 'software_version': '4.15', 'memSize': '7886', 'cpuSpeed': '2660.123', 'numaNodes': {u'0': {'totalMemory': '7886', 'cpus': [0, 1, 2, 3]}}, 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'pc-q35-1.4', u'pc-q35-1.5', u'q35', u'isapc', u'pc-0.10', u'pc-0.11', u'pc-0.12', u'pc-0.13', u'pc-0.14', u'pc-0.15', u'pc-1.0', u'pc-1.1', u'pc-1.2', u'pc-1.3', u'pc-i440fx-1.4', u'pc-i440fx-1.5', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '1', 'version': '20', 'name': 'Fedora'}, 'lastClient': '10.35.4.70'}} >Thread-19::DEBUG::2014-07-14 18:49:52,070::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call getHardwareInfo with () {} >Thread-19::DEBUG::2014-07-14 18:49:52,113::BindingXMLRPC::1114::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'OptiPlex 780', 'systemUUID': '44454c4c-3800-1056-8032-c8c04f34354a', 'systemSerialNumber': 'H8V245J', 'systemManufacturer': 'Dell Inc.'}} >Thread-19::DEBUG::2014-07-14 18:49:52,190::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call getCapabilities with () {} flowID [612c2306] >Thread-19::DEBUG::2014-07-14 18:49:52,192::utils::717::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) >Thread-19::DEBUG::2014-07-14 18:49:52,202::utils::737::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:49:52,228::BindingXMLRPC::1114::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:d2dedd7a362'}], 'FC': []}, 'packages2': {'kernel': {'release': '301.fc20.x86_64', 'buildtime': 1386244877.0, 'version': '3.11.10'}, 'spice-server': {'release': '3.fc20', 'buildtime': 1383129969L, 'version': '0.12.4'}, 'vdsm': {'release': '136.git02f4fd5.fc19', 'buildtime': 1403104797L, 'version': '4.15.0'}, 'qemu-kvm': {'release': '4.fc20', 'buildtime': 1398798279L, 'version': '1.6.2'}, 'libvirt': {'release': '4.fc20', 'buildtime': 1395169169L, 'version': '1.1.3.4'}, 'qemu-img': {'release': '4.fc20', 'buildtime': 1398798279L, 'version': '1.6.2'}, 'mom': {'release': '1.master.fc19', 'buildtime': 1403103991L, 'version': '0.4.0'}}, 'numaNodeDistance': {'0': [10]}, 'cpuModel': 'Intel(R) Core(TM)2 Quad CPU Q8400 @ 2.66GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'selinux': {'mode': '1'}, 'kdumpStatus': 0, 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '10.35.4.198', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'gateway': '10.35.5.254', 'bootproto4': 'dhcp', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv4addrs': ['10.35.4.198/23'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.4.198', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv4addrs': ['10.35.4.198/23'], 'ipv6gateway': '::', 'gateway': '10.35.5.254', 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '195', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.180373ae5fe3', 'bridge_id': '8000.180373ae5fe3', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '841', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'multicast_query_interval': '12500', 'multicast_query_use_ifaddr': '0', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}, 'ports': ['em1']}}, 'uuid': '44454C4C-3800-1056-8032-C8C04F34354A', 'lastClientIface': 'ovirtmgmt', 'nics': {'p2p2': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'p2p2', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '7ab5c17e-4d3d-4d9b-86ad-31e8e31eb4be', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': '00:07:E9:A5:1A:AF', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:07:e9:a5:1a:af', 'speed': 0}, 'em1': {'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '18:03:73:ae:5f:e3', 'DEVICE': 'em1', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::1a03:73ff:feae:5fe3/64'], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '18:03:73:ae:5f:e3', 'speed': 100}, 'p2p1': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'p2p1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '77c39be3-75f9-438a-b26a-ca3253e979fb', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': '00:07:E9:A5:1A:AE', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:07:e9:a5:1a:ae', 'speed': 0}}, 'software_revision': '136', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,xsave,lahf_lm,dtherm,tpr_shadow,vnmi,flexpriority,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:d2dedd7a362', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'autoNumaBalancing': 2, 'reservedMem': '321', 'bondings': {'bond0': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '5a:15:34:d5:9a:79', 'slaves': [], 'options': {}}}, 'software_version': '4.15', 'memSize': '7886', 'cpuSpeed': '2660.123', 'numaNodes': {u'0': {'totalMemory': '7886', 'cpus': [0, 1, 2, 3]}}, 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'pc-q35-1.4', u'pc-q35-1.5', u'q35', u'isapc', u'pc-0.10', u'pc-0.11', u'pc-0.12', u'pc-0.13', u'pc-0.14', u'pc-0.15', u'pc-1.0', u'pc-1.1', u'pc-1.2', u'pc-1.3', u'pc-i440fx-1.4', u'pc-i440fx-1.5', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '1', 'version': '20', 'name': 'Fedora'}, 'lastClient': '10.35.4.70'}} >Thread-19::DEBUG::2014-07-14 18:49:52,245::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call getHardwareInfo with () {} flowID [612c2306] >Thread-19::DEBUG::2014-07-14 18:49:52,245::BindingXMLRPC::1114::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'OptiPlex 780', 'systemUUID': '44454c4c-3800-1056-8032-c8c04f34354a', 'systemSerialNumber': 'H8V245J', 'systemManufacturer': 'Dell Inc.'}} >Thread-19::DEBUG::2014-07-14 18:49:52,354::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call vmGetStats with ('dc518130-d5f0-455f-b5eb-76f471f41cf4',) {} flowID [612c2306] >Thread-19::DEBUG::2014-07-14 18:49:52,354::BindingXMLRPC::1114::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Paused', 'displayInfo': [{'tlsPort': u'5900', 'ipAddress': '10.35.4.198', 'type': 'spice', 'port': '-1'}], 'hash': '-3259405644078031619', 'acpiEnable': 'true', 'pid': '6988', 'displayIp': '10.35.4.198', 'displayPort': '-1', 'displaySecurePort': u'5900', 'timeOffset': '0', 'kvmEnable': 'true', 'network': {}, 'vmId': 'dc518130-d5f0-455f-b5eb-76f471f41cf4', 'vmType': 'kvm', 'disks': {}, 'monitorResponse': '0', 'elapsedTime': '427820', 'displayType': 'qxl', 'clientIp': ''}]} >Thread-19::DEBUG::2014-07-14 18:49:52,441::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call setMOMPolicyParameters with ({'balloonEnabled': False, 'ksmEnabled': False},) {} flowID [54d1545b] >Thread-19::DEBUG::2014-07-14 18:49:52,527::BindingXMLRPC::1114::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}} >Thread-19::DEBUG::2014-07-14 18:49:55,920::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call vmGetStats with ('dc518130-d5f0-455f-b5eb-76f471f41cf4',) {} >Thread-19::DEBUG::2014-07-14 18:49:55,920::BindingXMLRPC::1114::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Paused', 'displayInfo': [{'tlsPort': u'5900', 'ipAddress': '10.35.4.198', 'type': 'spice', 'port': '-1'}], 'hash': '-3259405644078031619', 'acpiEnable': 'true', 'pid': '6988', 'displayIp': '10.35.4.198', 'displayPort': '-1', 'displaySecurePort': u'5900', 'timeOffset': '0', 'kvmEnable': 'true', 'network': {}, 'vmId': 'dc518130-d5f0-455f-b5eb-76f471f41cf4', 'vmType': 'kvm', 'disks': {}, 'monitorResponse': '0', 'elapsedTime': '427824', 'displayType': 'qxl', 'clientIp': ''}]} >clientIFinit::DEBUG::2014-07-14 18:49:56,691::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d442deb-fec5-44eb-b3af-acb346979724`::moving from state init -> state preparing >clientIFinit::INFO::2014-07-14 18:49:56,691::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >clientIFinit::INFO::2014-07-14 18:49:56,691::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:49:56,692::task::1191::Storage.TaskManager.Task::(prepare) Task=`7d442deb-fec5-44eb-b3af-acb346979724`::finished: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:49:56,692::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d442deb-fec5-44eb-b3af-acb346979724`::moving from state preparing -> state finished >clientIFinit::DEBUG::2014-07-14 18:49:56,692::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >clientIFinit::DEBUG::2014-07-14 18:49:56,692::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >clientIFinit::DEBUG::2014-07-14 18:49:56,692::task::993::Storage.TaskManager.Task::(_decref) Task=`7d442deb-fec5-44eb-b3af-acb346979724`::ref 0 aborting False >clientIFinit::DEBUG::2014-07-14 18:50:01,698::task::595::Storage.TaskManager.Task::(_updateState) Task=`9fe4b86e-baf7-4c51-8a2e-7f0bfebb71e6`::moving from state init -> state preparing >clientIFinit::INFO::2014-07-14 18:50:01,698::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >clientIFinit::INFO::2014-07-14 18:50:01,698::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:50:01,699::task::1191::Storage.TaskManager.Task::(prepare) Task=`9fe4b86e-baf7-4c51-8a2e-7f0bfebb71e6`::finished: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:50:01,699::task::595::Storage.TaskManager.Task::(_updateState) Task=`9fe4b86e-baf7-4c51-8a2e-7f0bfebb71e6`::moving from state preparing -> state finished >clientIFinit::DEBUG::2014-07-14 18:50:01,699::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >clientIFinit::DEBUG::2014-07-14 18:50:01,699::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >clientIFinit::DEBUG::2014-07-14 18:50:01,699::task::993::Storage.TaskManager.Task::(_decref) Task=`9fe4b86e-baf7-4c51-8a2e-7f0bfebb71e6`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:50:05,034::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a4d42b4-c32e-4cc8-b5b3-da557f5366dd`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:50:05,034::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:50:05,034::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-19::DEBUG::2014-07-14 18:50:05,035::task::1191::Storage.TaskManager.Task::(prepare) Task=`1a4d42b4-c32e-4cc8-b5b3-da557f5366dd`::finished: {} >Thread-19::DEBUG::2014-07-14 18:50:05,035::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a4d42b4-c32e-4cc8-b5b3-da557f5366dd`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:50:05,035::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:50:05,035::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:50:05,035::task::993::Storage.TaskManager.Task::(_decref) Task=`1a4d42b4-c32e-4cc8-b5b3-da557f5366dd`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:50:06,475::BindingXMLRPC::1107::vds::(wrapper) client [10.35.4.70]::call vmDestroy with ('dc518130-d5f0-455f-b5eb-76f471f41cf4',) {} flowID [7dfab1c8] >Thread-19::INFO::2014-07-14 18:50:06,475::API::333::vds::(destroy) vmContainerLock acquired by vm dc518130-d5f0-455f-b5eb-76f471f41cf4 >Thread-19::DEBUG::2014-07-14 18:50:06,476::vm::4839::vm.Vm::(destroy) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::destroy Called >Thread-19::INFO::2014-07-14 18:50:06,476::vm::4786::vm.Vm::(releaseVm) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Release VM resources >libvirtEventLoop::DEBUG::2014-07-14 18:50:06,694::vm::5376::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::event Stopped detail 1 opaque None >libvirtEventLoop::INFO::2014-07-14 18:50:06,695::vm::2319::vm.Vm::(_onQemuDeath) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::underlying process disconnected >Thread-19::DEBUG::2014-07-14 18:50:06,696::task::595::Storage.TaskManager.Task::(_updateState) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:50:06,696::logUtils::44::dispatcher::(wrapper) Run and protect: teardownImage(sdUUID='1f0449ec-49a1-4c13-b973-a24382d1db5b', spUUID='2d31dd81-c30f-45b1-8933-2aeeae6ca427', imgUUID='da775677-8785-45a4-a0de-0ad497e0adb6', volUUID=None) >Thread-19::DEBUG::2014-07-14 18:50:06,696::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b`ReqID=`ae5a5a84-e6c6-48cd-831b-16641e4b5ca5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3218' at 'teardownImage' >Thread-19::DEBUG::2014-07-14 18:50:06,697::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b' for lock type 'shared' >Thread-19::DEBUG::2014-07-14 18:50:06,697::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b' is free. Now locking as 'shared' (1 active user) >Thread-19::DEBUG::2014-07-14 18:50:06,697::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b`ReqID=`ae5a5a84-e6c6-48cd-831b-16641e4b5ca5`::Granted request >Thread-19::DEBUG::2014-07-14 18:50:06,697::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::_resourcesAcquired: Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b (shared) >Thread-19::DEBUG::2014-07-14 18:50:06,698::task::993::Storage.TaskManager.Task::(_decref) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::ref 1 aborting False >Thread-19::ERROR::2014-07-14 18:50:06,698::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-19::ERROR::2014-07-14 18:50:06,698::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1f0449ec-49a1-4c13-b973-a24382d1db5b >Thread-19::DEBUG::2014-07-14 18:50:06,698::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:50:06,699::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1f0449ec-49a1-4c13-b973-a24382d1db5b' (cwd None) >clientIFinit::DEBUG::2014-07-14 18:50:06,710::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f784afa-771d-4605-a37b-0fd2357d77dd`::moving from state init -> state preparing >clientIFinit::INFO::2014-07-14 18:50:06,711::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >clientIFinit::INFO::2014-07-14 18:50:06,711::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:50:06,711::task::1191::Storage.TaskManager.Task::(prepare) Task=`2f784afa-771d-4605-a37b-0fd2357d77dd`::finished: {'poollist': []} >clientIFinit::DEBUG::2014-07-14 18:50:06,711::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f784afa-771d-4605-a37b-0fd2357d77dd`::moving from state preparing -> state finished >clientIFinit::DEBUG::2014-07-14 18:50:06,711::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >clientIFinit::DEBUG::2014-07-14 18:50:06,712::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >clientIFinit::DEBUG::2014-07-14 18:50:06,719::task::993::Storage.TaskManager.Task::(_decref) Task=`2f784afa-771d-4605-a37b-0fd2357d77dd`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:50:06,732::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n Volume group "1f0449ec-49a1-4c13-b973-a24382d1db5b" not found\n Skipping volume group 1f0449ec-49a1-4c13-b973-a24382d1db5b\n'; <rc> = 5 >Thread-19::WARNING::2014-07-14 18:50:06,733::lvm::370::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', ' Volume group "1f0449ec-49a1-4c13-b973-a24382d1db5b" not found', ' Skipping volume group 1f0449ec-49a1-4c13-b973-a24382d1db5b'] >Thread-19::DEBUG::2014-07-14 18:50:06,733::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19::ERROR::2014-07-14 18:50:06,741::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1f0449ec-49a1-4c13-b973-a24382d1db5b not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('1f0449ec-49a1-4c13-b973-a24382d1db5b',) >Thread-19::ERROR::2014-07-14 18:50:06,741::task::866::Storage.TaskManager.Task::(_setError) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 3220, in teardownImage > dom = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('1f0449ec-49a1-4c13-b973-a24382d1db5b',) >Thread-19::DEBUG::2014-07-14 18:50:06,741::task::885::Storage.TaskManager.Task::(_run) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::Task._run: eebbb24a-e670-45f7-93bf-611763faf223 ('1f0449ec-49a1-4c13-b973-a24382d1db5b', '2d31dd81-c30f-45b1-8933-2aeeae6ca427', 'da775677-8785-45a4-a0de-0ad497e0adb6') {} failed - stopping task >Thread-19::DEBUG::2014-07-14 18:50:06,741::task::1217::Storage.TaskManager.Task::(stop) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::stopping in state preparing (force False) >Thread-19::DEBUG::2014-07-14 18:50:06,742::task::993::Storage.TaskManager.Task::(_decref) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::ref 1 aborting True >Thread-19::INFO::2014-07-14 18:50:06,742::task::1171::Storage.TaskManager.Task::(prepare) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 >Thread-19::DEBUG::2014-07-14 18:50:06,742::task::1176::Storage.TaskManager.Task::(prepare) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::Prepare: aborted: Storage domain does not exist >Thread-19::DEBUG::2014-07-14 18:50:06,742::task::993::Storage.TaskManager.Task::(_decref) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::ref 0 aborting True >Thread-19::DEBUG::2014-07-14 18:50:06,742::task::928::Storage.TaskManager.Task::(_doAbort) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::Task._doAbort: force False >Thread-19::DEBUG::2014-07-14 18:50:06,742::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:50:06,743::task::595::Storage.TaskManager.Task::(_updateState) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::moving from state preparing -> state aborting >Thread-19::DEBUG::2014-07-14 18:50:06,743::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::_aborting: recover policy none >Thread-19::DEBUG::2014-07-14 18:50:06,743::task::595::Storage.TaskManager.Task::(_updateState) Task=`eebbb24a-e670-45f7-93bf-611763faf223`::moving from state aborting -> state failed >Thread-19::DEBUG::2014-07-14 18:50:06,743::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b': < ResourceRef 'Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b', isValid: 'True' obj: 'None'>} >Thread-19::DEBUG::2014-07-14 18:50:06,743::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:50:06,743::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b' >Thread-19::DEBUG::2014-07-14 18:50:06,744::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b' (0 active users) >Thread-19::DEBUG::2014-07-14 18:50:06,744::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:50:06,744::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.1f0449ec-49a1-4c13-b973-a24382d1db5b', Clearing records. >Thread-19::ERROR::2014-07-14 18:50:06,744::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: ('1f0449ec-49a1-4c13-b973-a24382d1db5b',)", 'code': 358}} >Thread-19::DEBUG::2014-07-14 18:50:06,745::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef24a297-ef0a-4a74-a81e-30d1d9237dc9`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:50:06,745::logUtils::44::dispatcher::(wrapper) Run and protect: inappropriateDevices(thiefId='dc518130-d5f0-455f-b5eb-76f471f41cf4') >Thread-19::INFO::2014-07-14 18:50:06,746::logUtils::47::dispatcher::(wrapper) Run and protect: inappropriateDevices, Return response: None >Thread-19::DEBUG::2014-07-14 18:50:06,747::task::1191::Storage.TaskManager.Task::(prepare) Task=`ef24a297-ef0a-4a74-a81e-30d1d9237dc9`::finished: None >Thread-19::DEBUG::2014-07-14 18:50:06,747::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef24a297-ef0a-4a74-a81e-30d1d9237dc9`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:50:06,747::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:50:06,747::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:50:06,747::task::993::Storage.TaskManager.Task::(_decref) Task=`ef24a297-ef0a-4a74-a81e-30d1d9237dc9`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:50:06,748::vm::4833::vm.Vm::(deleteVm) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Total desktops after destroy of dc518130-d5f0-455f-b5eb-76f471f41cf4 is 0 >libvirtEventLoop::DEBUG::2014-07-14 18:50:06,748::vm::2763::vm.Vm::(setDownStatus) vmId=`dc518130-d5f0-455f-b5eb-76f471f41cf4`::Changed state to Down: Admin shut down from the engine (code=6) >Thread-19::DEBUG::2014-07-14 18:50:06,749::BindingXMLRPC::1114::vds::(wrapper) return vmDestroy with {'status': {'message': 'Machine destroyed', 'code': 0}} >Thread-19::DEBUG::2014-07-14 18:50:20,205::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a8bfa87-0dfc-4d6f-bccf-8d42ffaa8007`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:50:20,206::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:50:20,206::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-19::DEBUG::2014-07-14 18:50:20,206::task::1191::Storage.TaskManager.Task::(prepare) Task=`3a8bfa87-0dfc-4d6f-bccf-8d42ffaa8007`::finished: {} >Thread-19::DEBUG::2014-07-14 18:50:20,206::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a8bfa87-0dfc-4d6f-bccf-8d42ffaa8007`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:50:20,206::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:50:20,207::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:50:20,207::task::993::Storage.TaskManager.Task::(_decref) Task=`3a8bfa87-0dfc-4d6f-bccf-8d42ffaa8007`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:50:35,342::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a3b6818-a278-4561-949e-5f2bf00c3cdf`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:50:35,343::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:50:35,343::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-19::DEBUG::2014-07-14 18:50:35,343::task::1191::Storage.TaskManager.Task::(prepare) Task=`0a3b6818-a278-4561-949e-5f2bf00c3cdf`::finished: {} >Thread-19::DEBUG::2014-07-14 18:50:35,343::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a3b6818-a278-4561-949e-5f2bf00c3cdf`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:50:35,343::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:50:35,344::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:50:35,344::task::993::Storage.TaskManager.Task::(_decref) Task=`0a3b6818-a278-4561-949e-5f2bf00c3cdf`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:50:50,449::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e703a59-1fae-43f2-ace9-425c873ed796`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:50:50,450::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:50:50,450::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-19::DEBUG::2014-07-14 18:50:50,450::task::1191::Storage.TaskManager.Task::(prepare) Task=`5e703a59-1fae-43f2-ace9-425c873ed796`::finished: {} >Thread-19::DEBUG::2014-07-14 18:50:50,450::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e703a59-1fae-43f2-ace9-425c873ed796`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:50:50,450::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:50:50,450::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:50:50,450::task::993::Storage.TaskManager.Task::(_decref) Task=`5e703a59-1fae-43f2-ace9-425c873ed796`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:05,551::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e710d5a-e024-464b-af0e-f6c4fcbd71fe`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:05,551::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:51:05,551::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-19::DEBUG::2014-07-14 18:51:05,551::task::1191::Storage.TaskManager.Task::(prepare) Task=`5e710d5a-e024-464b-af0e-f6c4fcbd71fe`::finished: {} >Thread-19::DEBUG::2014-07-14 18:51:05,551::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e710d5a-e024-464b-af0e-f6c4fcbd71fe`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:05,552::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:51:05,552::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:05,552::task::993::Storage.TaskManager.Task::(_decref) Task=`5e710d5a-e024-464b-af0e-f6c4fcbd71fe`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:20,646::task::595::Storage.TaskManager.Task::(_updateState) Task=`2fd7c01f-cd08-46a6-a4a3-1d2cd653e8d5`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:20,647::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:51:20,647::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-19::DEBUG::2014-07-14 18:51:20,647::task::1191::Storage.TaskManager.Task::(prepare) Task=`2fd7c01f-cd08-46a6-a4a3-1d2cd653e8d5`::finished: {} >Thread-19::DEBUG::2014-07-14 18:51:20,647::task::595::Storage.TaskManager.Task::(_updateState) Task=`2fd7c01f-cd08-46a6-a4a3-1d2cd653e8d5`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:20,647::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:51:20,647::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:20,648::task::993::Storage.TaskManager.Task::(_decref) Task=`2fd7c01f-cd08-46a6-a4a3-1d2cd653e8d5`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:35,776::task::595::Storage.TaskManager.Task::(_updateState) Task=`df8524a6-48f4-4597-bfee-ebaafdceee2c`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:35,776::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:51:35,777::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-19::DEBUG::2014-07-14 18:51:35,777::task::1191::Storage.TaskManager.Task::(prepare) Task=`df8524a6-48f4-4597-bfee-ebaafdceee2c`::finished: {} >Thread-19::DEBUG::2014-07-14 18:51:35,777::task::595::Storage.TaskManager.Task::(_updateState) Task=`df8524a6-48f4-4597-bfee-ebaafdceee2c`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:35,777::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:51:35,777::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:35,777::task::993::Storage.TaskManager.Task::(_decref) Task=`df8524a6-48f4-4597-bfee-ebaafdceee2c`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:50,876::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef3b6c8f-2aeb-4979-a8e5-ada4be863489`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:50,877::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:51:50,877::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-19::DEBUG::2014-07-14 18:51:50,877::task::1191::Storage.TaskManager.Task::(prepare) Task=`ef3b6c8f-2aeb-4979-a8e5-ada4be863489`::finished: {} >Thread-19::DEBUG::2014-07-14 18:51:50,877::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef3b6c8f-2aeb-4979-a8e5-ada4be863489`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:50,877::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:51:50,877::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:50,878::task::993::Storage.TaskManager.Task::(_decref) Task=`ef3b6c8f-2aeb-4979-a8e5-ada4be863489`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:51,095::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [422c0979] >Thread-19::DEBUG::2014-07-14 18:51:51,096::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f50e928-c14d-40c2-a54c-d3cd71bd263d`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:51,096::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:51:51,103::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2 >Thread-19::DEBUG::2014-07-14 18:51:51,104::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.35.16.25:/volumes/smamit/frontend/gchaplik/data2 /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:51,273::hsm::2332::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2 >Thread-19::DEBUG::2014-07-14 18:51:51,275::hsm::2356::Storage.HSM::(__prefetchDomains) Found SD uuids: () >Thread-19::DEBUG::2014-07-14 18:51:51,276::hsm::2412::Storage.HSM::(connectStorageServer) knownSDs: {} >Thread-19::INFO::2014-07-14 18:51:51,276::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-19::DEBUG::2014-07-14 18:51:51,276::task::1191::Storage.TaskManager.Task::(prepare) Task=`8f50e928-c14d-40c2-a54c-d3cd71bd263d`::finished: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-19::DEBUG::2014-07-14 18:51:51,276::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f50e928-c14d-40c2-a54c-d3cd71bd263d`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:51,276::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:51:51,277::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:51,277::task::993::Storage.TaskManager.Task::(_decref) Task=`8f50e928-c14d-40c2-a54c-d3cd71bd263d`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:51,390::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [7972f4d5] >Thread-19::DEBUG::2014-07-14 18:51:51,390::task::595::Storage.TaskManager.Task::(_updateState) Task=`61e2146b-360a-4571-a3ff-571e7fe589fc`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:51,390::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:51:51,398::hsm::2332::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2 >Thread-19::DEBUG::2014-07-14 18:51:51,399::hsm::2356::Storage.HSM::(__prefetchDomains) Found SD uuids: () >Thread-19::DEBUG::2014-07-14 18:51:51,399::hsm::2412::Storage.HSM::(connectStorageServer) knownSDs: {} >Thread-19::INFO::2014-07-14 18:51:51,399::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d'}]} >Thread-19::DEBUG::2014-07-14 18:51:51,399::task::1191::Storage.TaskManager.Task::(prepare) Task=`61e2146b-360a-4571-a3ff-571e7fe589fc`::finished: {'statuslist': [{'status': 0, 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d'}]} >Thread-19::DEBUG::2014-07-14 18:51:51,400::task::595::Storage.TaskManager.Task::(_updateState) Task=`61e2146b-360a-4571-a3ff-571e7fe589fc`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:51,400::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:51:51,400::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:51,400::task::993::Storage.TaskManager.Task::(_decref) Task=`61e2146b-360a-4571-a3ff-571e7fe589fc`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:51,406::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [7972f4d5] >Thread-19::DEBUG::2014-07-14 18:51:51,407::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf2571e6-2979-4e44-9e67-351b32a6dbbd`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:51,407::logUtils::44::dispatcher::(wrapper) Run and protect: createStorageDomain(storageType=1, sdUUID='79818395-80e1-40af-adb6-328223a20ad9', domainName='data2', typeSpecificArg='10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', domClass=1, domVersion='3', options=None) >Thread-19::DEBUG::2014-07-14 18:51:51,407::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-19::DEBUG::2014-07-14 18:51:51,407::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:51:51,408::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-19::DEBUG::2014-07-14 18:51:51,408::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:51:51,408::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-19::DEBUG::2014-07-14 18:51:51,408::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:51,427::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:51:51,428::multipath::109::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /usr/sbin/multipath' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:51,488::multipath::109::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:51:51,489::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:51,489::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:51,489::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:51,489::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:51,490::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:51,490::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:51,490::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::ERROR::2014-07-14 18:51:51,490::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::ERROR::2014-07-14 18:51:51,490::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:51:51,491::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:51,492::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 79818395-80e1-40af-adb6-328223a20ad9' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:51,525::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n Volume group "79818395-80e1-40af-adb6-328223a20ad9" not found\n Skipping volume group 79818395-80e1-40af-adb6-328223a20ad9\n'; <rc> = 5 >Thread-19::WARNING::2014-07-14 18:51:51,527::lvm::370::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', ' Volume group "79818395-80e1-40af-adb6-328223a20ad9" not found', ' Skipping volume group 79818395-80e1-40af-adb6-328223a20ad9'] >Thread-19::DEBUG::2014-07-14 18:51:51,527::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19::ERROR::2014-07-14 18:51:51,637::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 79818395-80e1-40af-adb6-328223a20ad9 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('79818395-80e1-40af-adb6-328223a20ad9',) >Thread-19::INFO::2014-07-14 18:51:51,638::nfsSD::69::Storage.StorageDomain::(create) sdUUID=79818395-80e1-40af-adb6-328223a20ad9 domainName=data2 remotePath=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2 domClass=1 >Thread-19::DEBUG::2014-07-14 18:51:51,769::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-19::DEBUG::2014-07-14 18:51:51,770::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-19::WARNING::2014-07-14 18:51:51,770::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-19::DEBUG::2014-07-14 18:51:51,770::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-19::DEBUG::2014-07-14 18:51:51,770::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-19::DEBUG::2014-07-14 18:51:51,770::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Regular', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2a3037ad9a73c14db04f9f64ee7ad3963f4de08f'] >Thread-19::DEBUG::2014-07-14 18:51:51,774::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-19::DEBUG::2014-07-14 18:51:51,775::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:51:51,776::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-19::DEBUG::2014-07-14 18:51:51,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Regular', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2a3037ad9a73c14db04f9f64ee7ad3963f4de08f'] >Thread-19::DEBUG::2014-07-14 18:51:51,783::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-19::DEBUG::2014-07-14 18:51:51,783::resourceManager::421::Storage.ResourceManager::(registerNamespace) Registering namespace '79818395-80e1-40af-adb6-328223a20ad9_imageNS' >Thread-19::DEBUG::2014-07-14 18:51:51,783::resourceManager::421::Storage.ResourceManager::(registerNamespace) Registering namespace '79818395-80e1-40af-adb6-328223a20ad9_volumeNS' >Thread-19::DEBUG::2014-07-14 18:51:51,783::clusterlock::143::Storage.initSANLock::(initSANLock) Initializing SANLock for domain 79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:51:51,984::sd::433::Storage.StorageDomain::(initSPMlease) lease initialized successfully >Thread-19::DEBUG::2014-07-14 18:51:51,984::hsm::2651::Storage.HSM::(createStorageDomain) knownSDs: {79818395-80e1-40af-adb6-328223a20ad9: storage.nfsSD.findDomain} >Thread-19::INFO::2014-07-14 18:51:51,984::logUtils::47::dispatcher::(wrapper) Run and protect: createStorageDomain, Return response: None >Thread-19::DEBUG::2014-07-14 18:51:51,985::task::1191::Storage.TaskManager.Task::(prepare) Task=`bf2571e6-2979-4e44-9e67-351b32a6dbbd`::finished: None >Thread-19::DEBUG::2014-07-14 18:51:51,985::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf2571e6-2979-4e44-9e67-351b32a6dbbd`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:51,985::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:51:51,985::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:51,985::task::993::Storage.TaskManager.Task::(_decref) Task=`bf2571e6-2979-4e44-9e67-351b32a6dbbd`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:51,991::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [7972f4d5] >Thread-19::DEBUG::2014-07-14 18:51:51,991::task::595::Storage.TaskManager.Task::(_updateState) Task=`a250894c-3444-4fe0-b131-fd9f591ee8fb`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:51,991::logUtils::44::dispatcher::(wrapper) Run and protect: getStorageDomainStats(sdUUID='79818395-80e1-40af-adb6-328223a20ad9', options=None) >Thread-19::DEBUG::2014-07-14 18:51:51,992::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`959b77a7-a5a1-45bc-980f-c4ac3a613af2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2790' at 'getStorageDomainStats' >Thread-19::DEBUG::2014-07-14 18:51:51,992::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' for lock type 'shared' >Thread-19::DEBUG::2014-07-14 18:51:51,992::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free. Now locking as 'shared' (1 active user) >Thread-19::DEBUG::2014-07-14 18:51:51,992::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`959b77a7-a5a1-45bc-980f-c4ac3a613af2`::Granted request >Thread-19::DEBUG::2014-07-14 18:51:51,993::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a250894c-3444-4fe0-b131-fd9f591ee8fb`::_resourcesAcquired: Storage.79818395-80e1-40af-adb6-328223a20ad9 (shared) >Thread-19::DEBUG::2014-07-14 18:51:51,993::task::993::Storage.TaskManager.Task::(_decref) Task=`a250894c-3444-4fe0-b131-fd9f591ee8fb`::ref 1 aborting False >Thread-19::INFO::2014-07-14 18:51:51,994::logUtils::47::dispatcher::(wrapper) Run and protect: getStorageDomainStats, Return response: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '417153941504', 'disktotal': '536870912000', 'mdafree': 0}} >Thread-19::DEBUG::2014-07-14 18:51:51,994::task::1191::Storage.TaskManager.Task::(prepare) Task=`a250894c-3444-4fe0-b131-fd9f591ee8fb`::finished: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '417153941504', 'disktotal': '536870912000', 'mdafree': 0}} >Thread-19::DEBUG::2014-07-14 18:51:51,994::task::595::Storage.TaskManager.Task::(_updateState) Task=`a250894c-3444-4fe0-b131-fd9f591ee8fb`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:51,995::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.79818395-80e1-40af-adb6-328223a20ad9': < ResourceRef 'Storage.79818395-80e1-40af-adb6-328223a20ad9', isValid: 'True' obj: 'None'>} >Thread-19::DEBUG::2014-07-14 18:51:51,995::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:51,995::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' >Thread-19::DEBUG::2014-07-14 18:51:51,995::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' (0 active users) >Thread-19::DEBUG::2014-07-14 18:51:51,995::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:51:51,995::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9', Clearing records. >Thread-19::DEBUG::2014-07-14 18:51:51,996::task::993::Storage.TaskManager.Task::(_decref) Task=`a250894c-3444-4fe0-b131-fd9f591ee8fb`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:52,084::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [59a2146e] >Thread-19::DEBUG::2014-07-14 18:51:52,084::task::595::Storage.TaskManager.Task::(_updateState) Task=`dec50df4-f375-42e6-a6fd-0255c510d16b`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:52,084::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:51:52,085::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:52,120::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-19::DEBUG::2014-07-14 18:51:52,120::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:51:52,120::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-19::DEBUG::2014-07-14 18:51:52,120::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:51:52,121::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-19::DEBUG::2014-07-14 18:51:52,121::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:52,138::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:51:52,139::multipath::109::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /usr/sbin/multipath' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:52,169::multipath::109::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:51:52,170::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,170::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,170::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,170::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,171::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,171::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,171::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::INFO::2014-07-14 18:51:52,171::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d'}]} >Thread-19::DEBUG::2014-07-14 18:51:52,171::task::1191::Storage.TaskManager.Task::(prepare) Task=`dec50df4-f375-42e6-a6fd-0255c510d16b`::finished: {'statuslist': [{'status': 0, 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d'}]} >Thread-19::DEBUG::2014-07-14 18:51:52,172::task::595::Storage.TaskManager.Task::(_updateState) Task=`dec50df4-f375-42e6-a6fd-0255c510d16b`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:52,172::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:51:52,172::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:52,172::task::993::Storage.TaskManager.Task::(_decref) Task=`dec50df4-f375-42e6-a6fd-0255c510d16b`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:52,325::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [3b504545] >Thread-19::DEBUG::2014-07-14 18:51:52,325::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a741566-ca10-4589-92e5-d23650beea07`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:52,325::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:51:52,333::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2 >Thread-19::DEBUG::2014-07-14 18:51:52,333::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.35.16.25:/volumes/smamit/frontend/gchaplik/data2 /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:52,382::hsm::2332::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2 >Thread-19::DEBUG::2014-07-14 18:51:52,384::hsm::2356::Storage.HSM::(__prefetchDomains) Found SD uuids: ('79818395-80e1-40af-adb6-328223a20ad9',) >Thread-19::DEBUG::2014-07-14 18:51:52,385::hsm::2412::Storage.HSM::(connectStorageServer) knownSDs: {79818395-80e1-40af-adb6-328223a20ad9: storage.nfsSD.findDomain} >Thread-19::INFO::2014-07-14 18:51:52,385::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d'}]} >Thread-19::DEBUG::2014-07-14 18:51:52,385::task::1191::Storage.TaskManager.Task::(prepare) Task=`0a741566-ca10-4589-92e5-d23650beea07`::finished: {'statuslist': [{'status': 0, 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d'}]} >Thread-19::DEBUG::2014-07-14 18:51:52,385::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a741566-ca10-4589-92e5-d23650beea07`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:51:52,385::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:51:52,385::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:51:52,386::task::993::Storage.TaskManager.Task::(_decref) Task=`0a741566-ca10-4589-92e5-d23650beea07`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:51:52,393::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [3b504545] >Thread-19::DEBUG::2014-07-14 18:51:52,394::task::595::Storage.TaskManager.Task::(_updateState) Task=`82d9cfb2-d2ad-427f-bb0c-baf3d6c14ac5`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:51:52,394::logUtils::44::dispatcher::(wrapper) Run and protect: createStoragePool(poolType=None, spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', poolName='dc34', masterDom='79818395-80e1-40af-adb6-328223a20ad9', domList=['79818395-80e1-40af-adb6-328223a20ad9'], masterVersion=1, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, options=None) >Thread-19::DEBUG::2014-07-14 18:51:52,394::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-19::DEBUG::2014-07-14 18:51:52,394::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:51:52,395::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-19::DEBUG::2014-07-14 18:51:52,395::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:51:52,395::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-19::DEBUG::2014-07-14 18:51:52,395::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:52,413::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:51:52,414::multipath::109::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /usr/sbin/multipath' (cwd None) >Thread-19::DEBUG::2014-07-14 18:51:52,444::multipath::109::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:51:52,444::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,444::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,445::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,445::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,445::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,445::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:51:52,445::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:51:52,462::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:51:52,463::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-19::DEBUG::2014-07-14 18:51:52,466::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Regular', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2a3037ad9a73c14db04f9f64ee7ad3963f4de08f'] >Thread-19::DEBUG::2014-07-14 18:51:52,469::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-19::INFO::2014-07-14 18:51:52,469::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 79818395-80e1-40af-adb6-328223a20ad9_imageNS already registered >Thread-19::INFO::2014-07-14 18:51:52,469::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 79818395-80e1-40af-adb6-328223a20ad9_volumeNS already registered >Thread-19::DEBUG::2014-07-14 18:51:52,470::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`2c82c35f-b0bd-4230-85ac-ae57170249ce`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '943' at 'createStoragePool' >Thread-19::DEBUG::2014-07-14 18:51:52,470::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'exclusive' >Thread-19::DEBUG::2014-07-14 18:51:52,470::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'exclusive' (1 active user) >Thread-19::DEBUG::2014-07-14 18:51:52,470::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`2c82c35f-b0bd-4230-85ac-ae57170249ce`::Granted request >Thread-19::DEBUG::2014-07-14 18:51:52,471::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`82d9cfb2-d2ad-427f-bb0c-baf3d6c14ac5`::_resourcesAcquired: Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c (exclusive) >Thread-19::DEBUG::2014-07-14 18:51:52,471::task::993::Storage.TaskManager.Task::(_decref) Task=`82d9cfb2-d2ad-427f-bb0c-baf3d6c14ac5`::ref 1 aborting False >Thread-19::DEBUG::2014-07-14 18:51:52,471::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`5a054af8-4b4a-4e9d-9ad7-4ba8f92a2ce6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '945' at 'createStoragePool' >Thread-19::DEBUG::2014-07-14 18:51:52,471::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' for lock type 'exclusive' >Thread-19::DEBUG::2014-07-14 18:51:52,472::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free. Now locking as 'exclusive' (1 active user) >Thread-19::DEBUG::2014-07-14 18:51:52,472::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`5a054af8-4b4a-4e9d-9ad7-4ba8f92a2ce6`::Granted request >Thread-19::DEBUG::2014-07-14 18:51:52,472::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`82d9cfb2-d2ad-427f-bb0c-baf3d6c14ac5`::_resourcesAcquired: Storage.79818395-80e1-40af-adb6-328223a20ad9 (exclusive) >Thread-19::DEBUG::2014-07-14 18:51:52,472::task::993::Storage.TaskManager.Task::(_decref) Task=`82d9cfb2-d2ad-427f-bb0c-baf3d6c14ac5`::ref 1 aborting False >Thread-19::INFO::2014-07-14 18:51:52,472::sp::133::Storage.StoragePool::(setBackend) updating pool 68055dba-2ea1-4ff1-864e-7c15d4b1334c backend from type NoneType instance 0x3a0e38ff00 to type StoragePoolDiskBackend instance 0x7f3e98718908 >Thread-19::INFO::2014-07-14 18:51:52,473::sp::545::Storage.StoragePool::(create) spUUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c poolName=dc34 master_sd=79818395-80e1-40af-adb6-328223a20ad9 domList=['79818395-80e1-40af-adb6-328223a20ad9'] masterVersion=1 {'LEASETIMESEC': 60, 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5} >Thread-19::INFO::2014-07-14 18:51:52,473::fileSD::350::Storage.StorageDomain::(validate) sdUUID=79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:51:52,475::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Regular', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2a3037ad9a73c14db04f9f64ee7ad3963f4de08f'] >Thread-19::DEBUG::2014-07-14 18:51:52,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Regular', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2a3037ad9a73c14db04f9f64ee7ad3963f4de08f'] >Thread-19::DEBUG::2014-07-14 18:51:52,479::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c >Thread-19::DEBUG::2014-07-14 18:51:52,480::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-19::DEBUG::2014-07-14 18:51:52,480::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-19::INFO::2014-07-14 18:51:52,480::clusterlock::183::Storage.SANLock::(acquireHostId) Acquiring host id for domain 79818395-80e1-40af-adb6-328223a20ad9 (id: 250) >Detector thread::DEBUG::2014-07-14 18:51:53,943::protocoldetector::159::vds.MultiProtocolAcceptor::(_add_connection) Adding connection from 10.35.4.70:51332 >Detector thread::DEBUG::2014-07-14 18:51:53,944::protocoldetector::170::vds.MultiProtocolAcceptor::(_remove_connection) Connection removed from 10.35.4.70:51332 >Detector thread::DEBUG::2014-07-14 18:51:53,944::protocoldetector::196::vds.MultiProtocolAcceptor::(_handle_connection_read) Detected protocol xml from 10.35.4.70:51332 >Detector thread::DEBUG::2014-07-14 18:51:53,944::BindingXMLRPC::1147::XmlDetector::(handleSocket) xml over http detected from ('10.35.4.70', 51332) >Thread-34::DEBUG::2014-07-14 18:52:05,976::task::595::Storage.TaskManager.Task::(_updateState) Task=`9835b8c1-f4a2-4743-bc59-faecdc9fba90`::moving from state init -> state preparing >Thread-34::INFO::2014-07-14 18:52:05,976::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34::INFO::2014-07-14 18:52:05,976::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-34::DEBUG::2014-07-14 18:52:05,977::task::1191::Storage.TaskManager.Task::(prepare) Task=`9835b8c1-f4a2-4743-bc59-faecdc9fba90`::finished: {} >Thread-34::DEBUG::2014-07-14 18:52:05,977::task::595::Storage.TaskManager.Task::(_updateState) Task=`9835b8c1-f4a2-4743-bc59-faecdc9fba90`::moving from state preparing -> state finished >Thread-34::DEBUG::2014-07-14 18:52:05,977::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34::DEBUG::2014-07-14 18:52:05,977::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34::DEBUG::2014-07-14 18:52:05,977::task::993::Storage.TaskManager.Task::(_decref) Task=`9835b8c1-f4a2-4743-bc59-faecdc9fba90`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:13,483::clusterlock::201::Storage.SANLock::(acquireHostId) Host id for domain 79818395-80e1-40af-adb6-328223a20ad9 successfully acquired (id: 250) >Thread-19::INFO::2014-07-14 18:52:13,483::clusterlock::234::Storage.SANLock::(acquire) Acquiring cluster lock for domain 79818395-80e1-40af-adb6-328223a20ad9 (id: 250) >Thread-19::DEBUG::2014-07-14 18:52:13,762::clusterlock::260::Storage.SANLock::(acquire) Cluster lock for domain 79818395-80e1-40af-adb6-328223a20ad9 successfully acquired (id: 250) >Thread-19::INFO::2014-07-14 18:52:13,763::sp::678::Storage.StoragePool::(createMaster) setting master domain for spUUID 68055dba-2ea1-4ff1-864e-7c15d4b1334c on sdUUID=79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:13,763::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-19::DEBUG::2014-07-14 18:52:13,763::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-19::DEBUG::2014-07-14 18:52:13,763::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Regular', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=62e5110ca2a4ac67f9fe8124e2b7154f0d63f287'] >Thread-19::DEBUG::2014-07-14 18:52:13,768::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-19::DEBUG::2014-07-14 18:52:13,771::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Regular', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=62e5110ca2a4ac67f9fe8124e2b7154f0d63f287'] >Thread-19::DEBUG::2014-07-14 18:52:13,772::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-19::DEBUG::2014-07-14 18:52:13,772::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-19::DEBUG::2014-07-14 18:52:13,772::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-19::DEBUG::2014-07-14 18:52:13,772::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-19::DEBUG::2014-07-14 18:52:13,773::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-19::DEBUG::2014-07-14 18:52:13,773::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be4cd943882c2df4da89ac9d42030969827cd416'] >Thread-19::DEBUG::2014-07-14 18:52:13,776::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-19::DEBUG::2014-07-14 18:52:13,777::sp::1387::Storage.StoragePool::(setMasterDomain) Master domain 79818395-80e1-40af-adb6-328223a20ad9 verified, version 1 >Thread-19::DEBUG::2014-07-14 18:52:13,777::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-19::DEBUG::2014-07-14 18:52:13,777::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:52:13,781::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be4cd943882c2df4da89ac9d42030969827cd416'] >Thread-19::INFO::2014-07-14 18:52:13,781::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 79818395-80e1-40af-adb6-328223a20ad9 >Thread-35::DEBUG::2014-07-14 18:52:13,782::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:13,782::sp::1410::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `68055dba-2ea1-4ff1-864e-7c15d4b1334c` started monitoring domain `79818395-80e1-40af-adb6-328223a20ad9` >Thread-19::DEBUG::2014-07-14 18:52:13,784::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:52:13,784::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c >Thread-19::WARNING::2014-07-14 18:52:13,784::fileUtils::149::Storage.fileUtils::(createdir) Dir /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c already exists >Thread-35::DEBUG::2014-07-14 18:52:13,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-19::DEBUG::2014-07-14 18:52:13,801::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be4cd943882c2df4da89ac9d42030969827cd416'] >Thread-35::DEBUG::2014-07-14 18:52:13,804::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n481 bytes (481 B) copied, 0.000668874 s, 719 kB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:13,805::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-35::DEBUG::2014-07-14 18:52:13,814::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 79818395-80e1-40af-adb6-328223a20ad9 changed its status to Valid >Thread-19::DEBUG::2014-07-14 18:52:13,818::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-19::DEBUG::2014-07-14 18:52:13,848::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n No volume groups found\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:13,849::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19::INFO::2014-07-14 18:52:13,857::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:13,857::sp::1135::Storage.StoragePool::(_linkStorageDomain) Creating symlink from /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::INFO::2014-07-14 18:52:13,857::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd >Thread-19::DEBUG::2014-07-14 18:52:13,858::sp::1135::Storage.StoragePool::(_linkStorageDomain) Creating symlink from /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd >Thread-19::INFO::2014-07-14 18:52:13,858::clusterlock::278::Storage.SANLock::(release) Releasing cluster lock for domain 79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:13,860::clusterlock::288::Storage.SANLock::(release) Cluster lock for domain 79818395-80e1-40af-adb6-328223a20ad9 successfully released >Thread-19::INFO::2014-07-14 18:52:13,860::clusterlock::206::Storage.SANLock::(releaseHostId) Releasing host id for domain 79818395-80e1-40af-adb6-328223a20ad9 (id: 250) >Thread-19::DEBUG::2014-07-14 18:52:15,861::clusterlock::216::Storage.SANLock::(releaseHostId) Host id for domain 79818395-80e1-40af-adb6-328223a20ad9 released successfully (id: 250) >Thread-19::INFO::2014-07-14 18:52:15,861::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 79818395-80e1-40af-adb6-328223a20ad9 >Thread-35::DEBUG::2014-07-14 18:52:15,861::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 79818395-80e1-40af-adb6-328223a20ad9 >Thread-35::INFO::2014-07-14 18:52:15,862::clusterlock::206::Storage.SANLock::(releaseHostId) Releasing host id for domain 79818395-80e1-40af-adb6-328223a20ad9 (id: 250) >Thread-35::DEBUG::2014-07-14 18:52:15,862::clusterlock::216::Storage.SANLock::(releaseHostId) Host id for domain 79818395-80e1-40af-adb6-328223a20ad9 released successfully (id: 250) >Thread-19::INFO::2014-07-14 18:52:15,862::logUtils::47::dispatcher::(wrapper) Run and protect: createStoragePool, Return response: True >Thread-19::DEBUG::2014-07-14 18:52:15,862::task::1191::Storage.TaskManager.Task::(prepare) Task=`82d9cfb2-d2ad-427f-bb0c-baf3d6c14ac5`::finished: True >Thread-19::DEBUG::2014-07-14 18:52:15,863::task::595::Storage.TaskManager.Task::(_updateState) Task=`82d9cfb2-d2ad-427f-bb0c-baf3d6c14ac5`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:15,863::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c': < ResourceRef 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', isValid: 'True' obj: 'None'>, 'Storage.79818395-80e1-40af-adb6-328223a20ad9': < ResourceRef 'Storage.79818395-80e1-40af-adb6-328223a20ad9', isValid: 'True' obj: 'None'>} >Thread-19::DEBUG::2014-07-14 18:52:15,863::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:15,863::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-19::DEBUG::2014-07-14 18:52:15,863::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-19::DEBUG::2014-07-14 18:52:15,863::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:52:15,864::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-19::DEBUG::2014-07-14 18:52:15,864::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' >Thread-19::DEBUG::2014-07-14 18:52:15,864::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' (0 active users) >Thread-19::DEBUG::2014-07-14 18:52:15,864::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:52:15,864::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9', Clearing records. >Thread-19::DEBUG::2014-07-14 18:52:15,865::task::993::Storage.TaskManager.Task::(_decref) Task=`82d9cfb2-d2ad-427f-bb0c-baf3d6c14ac5`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:15,896::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [17c05dbe] >Thread-19::DEBUG::2014-07-14 18:52:15,896::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb6dd564-e324-4fb5-8081-b1bd287c6de0`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:15,896::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:52:15,904::hsm::2332::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2 >Thread-19::DEBUG::2014-07-14 18:52:15,905::hsm::2356::Storage.HSM::(__prefetchDomains) Found SD uuids: ('79818395-80e1-40af-adb6-328223a20ad9',) >Thread-19::DEBUG::2014-07-14 18:52:15,905::hsm::2412::Storage.HSM::(connectStorageServer) knownSDs: {79818395-80e1-40af-adb6-328223a20ad9: storage.nfsSD.findDomain} >Thread-19::INFO::2014-07-14 18:52:15,905::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d'}]} >Thread-19::DEBUG::2014-07-14 18:52:15,905::task::1191::Storage.TaskManager.Task::(prepare) Task=`bb6dd564-e324-4fb5-8081-b1bd287c6de0`::finished: {'statuslist': [{'status': 0, 'id': '37bde018-f2d8-4fc6-a835-4481cfee019d'}]} >Thread-19::DEBUG::2014-07-14 18:52:15,905::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb6dd564-e324-4fb5-8081-b1bd287c6de0`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:15,905::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:15,906::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:15,906::task::993::Storage.TaskManager.Task::(_decref) Task=`bb6dd564-e324-4fb5-8081-b1bd287c6de0`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:15,947::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:15,947::task::595::Storage.TaskManager.Task::(_updateState) Task=`05f94112-848c-4689-a302-d476bfda98ad`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:15,947::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-19::ERROR::2014-07-14 18:52:15,948::task::866::Storage.TaskManager.Task::(_setError) Task=`05f94112-848c-4689-a302-d476bfda98ad`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 607, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('68055dba-2ea1-4ff1-864e-7c15d4b1334c',) >Thread-19::DEBUG::2014-07-14 18:52:15,948::task::885::Storage.TaskManager.Task::(_run) Task=`05f94112-848c-4689-a302-d476bfda98ad`::Task._run: 05f94112-848c-4689-a302-d476bfda98ad ('68055dba-2ea1-4ff1-864e-7c15d4b1334c',) {} failed - stopping task >Thread-19::DEBUG::2014-07-14 18:52:15,948::task::1217::Storage.TaskManager.Task::(stop) Task=`05f94112-848c-4689-a302-d476bfda98ad`::stopping in state preparing (force False) >Thread-19::DEBUG::2014-07-14 18:52:15,948::task::993::Storage.TaskManager.Task::(_decref) Task=`05f94112-848c-4689-a302-d476bfda98ad`::ref 1 aborting True >Thread-19::INFO::2014-07-14 18:52:15,949::task::1171::Storage.TaskManager.Task::(prepare) Task=`05f94112-848c-4689-a302-d476bfda98ad`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-19::DEBUG::2014-07-14 18:52:15,949::task::1176::Storage.TaskManager.Task::(prepare) Task=`05f94112-848c-4689-a302-d476bfda98ad`::Prepare: aborted: Unknown pool id, pool not connected >Thread-19::DEBUG::2014-07-14 18:52:15,949::task::993::Storage.TaskManager.Task::(_decref) Task=`05f94112-848c-4689-a302-d476bfda98ad`::ref 0 aborting True >Thread-19::DEBUG::2014-07-14 18:52:15,949::task::928::Storage.TaskManager.Task::(_doAbort) Task=`05f94112-848c-4689-a302-d476bfda98ad`::Task._doAbort: force False >Thread-19::DEBUG::2014-07-14 18:52:15,949::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:15,949::task::595::Storage.TaskManager.Task::(_updateState) Task=`05f94112-848c-4689-a302-d476bfda98ad`::moving from state preparing -> state aborting >Thread-19::DEBUG::2014-07-14 18:52:15,949::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`05f94112-848c-4689-a302-d476bfda98ad`::_aborting: recover policy none >Thread-19::DEBUG::2014-07-14 18:52:15,950::task::595::Storage.TaskManager.Task::(_updateState) Task=`05f94112-848c-4689-a302-d476bfda98ad`::moving from state aborting -> state failed >Thread-19::DEBUG::2014-07-14 18:52:15,950::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:15,950::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::ERROR::2014-07-14 18:52:15,950::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Unknown pool id, pool not connected: ('68055dba-2ea1-4ff1-864e-7c15d4b1334c',)", 'code': 309}} >Thread-19::DEBUG::2014-07-14 18:52:15,958::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:15,958::task::595::Storage.TaskManager.Task::(_updateState) Task=`15c0a08d-dab4-471b-95a7-513ef09685c7`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:15,958::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', hostID=1, msdUUID='79818395-80e1-40af-adb6-328223a20ad9', masterVersion=1, domainsMap=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:15,959::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`be315491-ab11-4a2c-af1a-a9a277591404`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '981' at 'connectStoragePool' >Thread-19::DEBUG::2014-07-14 18:52:15,959::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-19::DEBUG::2014-07-14 18:52:15,959::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-19::DEBUG::2014-07-14 18:52:15,959::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`be315491-ab11-4a2c-af1a-a9a277591404`::Granted request >Thread-19::DEBUG::2014-07-14 18:52:15,960::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`b60033b7-d81f-44e8-87d9-b2d2bd3d58b1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1029' at '_connectStoragePool' >Thread-19::DEBUG::2014-07-14 18:52:15,960::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'exclusive' >Thread-19::DEBUG::2014-07-14 18:52:15,960::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'exclusive' (1 active user) >Thread-19::DEBUG::2014-07-14 18:52:15,960::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`b60033b7-d81f-44e8-87d9-b2d2bd3d58b1`::Granted request >Thread-19::INFO::2014-07-14 18:52:15,961::sp::133::Storage.StoragePool::(setBackend) updating pool 68055dba-2ea1-4ff1-864e-7c15d4b1334c backend from type NoneType instance 0x3a0e38ff00 to type StoragePoolDiskBackend instance 0x7f3e98718638 >Thread-19::INFO::2014-07-14 18:52:15,961::sp::630::Storage.StoragePool::(connect) Connect host #1 to the storage pool 68055dba-2ea1-4ff1-864e-7c15d4b1334c with master domain: 79818395-80e1-40af-adb6-328223a20ad9 (ver = 1) >Thread-19::DEBUG::2014-07-14 18:52:15,961::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:15,961::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:15,962::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:15,962::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:15,962::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:15,962::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:15,962::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-19::DEBUG::2014-07-14 18:52:15,963::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:52:15,963::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-19::DEBUG::2014-07-14 18:52:15,963::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:52:15,963::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-19::DEBUG::2014-07-14 18:52:15,963::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R' (cwd None) >Thread-19::DEBUG::2014-07-14 18:52:15,981::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:52:15,981::multipath::109::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /usr/sbin/multipath' (cwd None) >Thread-19::DEBUG::2014-07-14 18:52:16,012::multipath::109::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:16,012::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:16,012::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:16,013::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:16,013::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:16,013::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:16,013::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:16,013::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:52:16,031::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:16,033::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-19::DEBUG::2014-07-14 18:52:16,036::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be4cd943882c2df4da89ac9d42030969827cd416'] >Thread-19::DEBUG::2014-07-14 18:52:16,037::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-19::INFO::2014-07-14 18:52:16,038::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 79818395-80e1-40af-adb6-328223a20ad9_imageNS already registered >Thread-19::INFO::2014-07-14 18:52:16,038::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 79818395-80e1-40af-adb6-328223a20ad9_volumeNS already registered >Thread-19::DEBUG::2014-07-14 18:52:16,038::sp::1387::Storage.StoragePool::(setMasterDomain) Master domain 79818395-80e1-40af-adb6-328223a20ad9 verified, version 1 >Thread-19::DEBUG::2014-07-14 18:52:16,038::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-19::DEBUG::2014-07-14 18:52:16,038::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:52:16,041::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be4cd943882c2df4da89ac9d42030969827cd416'] >Thread-19::INFO::2014-07-14 18:52:16,042::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:16,042::sp::1410::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `68055dba-2ea1-4ff1-864e-7c15d4b1334c` started monitoring domain `79818395-80e1-40af-adb6-328223a20ad9` >Thread-46::DEBUG::2014-07-14 18:52:16,043::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:16,043::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:52:16,044::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c >Thread-19::WARNING::2014-07-14 18:52:16,045::fileUtils::149::Storage.fileUtils::(createdir) Dir /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c already exists >Thread-46::DEBUG::2014-07-14 18:52:16,057::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-07-14 18:52:16,070::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n481 bytes (481 B) copied, 0.000700515 s, 687 kB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:16,149::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be4cd943882c2df4da89ac9d42030969827cd416'] >Thread-19::DEBUG::2014-07-14 18:52:16,151::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-46::DEBUG::2014-07-14 18:52:16,152::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 79818395-80e1-40af-adb6-328223a20ad9 changed its status to Valid >Thread-47::DEBUG::2014-07-14 18:52:16,154::misc::816::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-46::INFO::2014-07-14 18:52:16,155::clusterlock::183::Storage.SANLock::(acquireHostId) Acquiring host id for domain 79818395-80e1-40af-adb6-328223a20ad9 (id: 1) >Thread-47::DEBUG::2014-07-14 18:52:16,157::misc::826::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-46::DEBUG::2014-07-14 18:52:16,158::clusterlock::201::Storage.SANLock::(acquireHostId) Host id for domain 79818395-80e1-40af-adb6-328223a20ad9 successfully acquired (id: 1) >Thread-47::DEBUG::2014-07-14 18:52:16,161::misc::836::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-19::DEBUG::2014-07-14 18:52:16,162::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-48::INFO::2014-07-14 18:52:16,173::clientIF::137::vds::(contEIOVms) vmContainerLock acquired >Thread-19::DEBUG::2014-07-14 18:52:16,190::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n No volume groups found\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:16,190::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19::INFO::2014-07-14 18:52:16,199::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:16,199::sp::1135::Storage.StoragePool::(_linkStorageDomain) Creating symlink from /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::INFO::2014-07-14 18:52:16,200::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd >Thread-19::DEBUG::2014-07-14 18:52:16,200::sp::1135::Storage.StoragePool::(_linkStorageDomain) Creating symlink from /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd >Thread-19::DEBUG::2014-07-14 18:52:16,201::threadPool::35::Storage.ThreadPool::(__init__) Enter - numThreads: 5, waitTimeout: 3, maxTasks: 500 >Thread-19::DEBUG::2014-07-14 18:52:16,205::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) '/usr/bin/dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8' (cwd None) >Thread-19::DEBUG::2014-07-14 18:52:16,221::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) SUCCESS: <err> = '8+0 records in\n8+0 records out\n4096 bytes (4.1 kB) copied, 0.00457355 s, 896 kB/s\n'; <rc> = 0 >Thread-19::INFO::2014-07-14 18:52:16,222::storage_mailbox::385::Storage.MailBox.HsmMailMonitor::(_sendMail) HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=512', 'seek=8'] >Thread-19::DEBUG::2014-07-14 18:52:16,222::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) '/usr/bin/dd of=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=fullblock oflag=direct conv=notrunc bs=512 seek=8' (cwd None) >Thread-19::DEBUG::2014-07-14 18:52:16,241::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) SUCCESS: <err> = '8+0 records in\n8+0 records out\n4096 bytes (4.1 kB) copied, 0.00670876 s, 611 kB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:16,242::storage_mailbox::207::Storage.Mailbox.HSM::(__init__) HSM_MailboxMonitor created for pool 68055dba-2ea1-4ff1-864e-7c15d4b1334c >Thread-19::DEBUG::2014-07-14 18:52:16,242::sp::462::Storage.StoragePool::(__createMailboxMonitor) HSM mailbox ready for pool 68055dba-2ea1-4ff1-864e-7c15d4b1334c on master domain 79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:16,242::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-19::DEBUG::2014-07-14 18:52:16,243::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-19::DEBUG::2014-07-14 18:52:16,243::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:52:16,243::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-19::DEBUG::2014-07-14 18:52:16,243::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-19::DEBUG::2014-07-14 18:52:16,243::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-19::DEBUG::2014-07-14 18:52:16,243::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:52:16,244::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-19::INFO::2014-07-14 18:52:16,244::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-19::DEBUG::2014-07-14 18:52:16,244::task::1191::Storage.TaskManager.Task::(prepare) Task=`15c0a08d-dab4-471b-95a7-513ef09685c7`::finished: True >Thread-19::DEBUG::2014-07-14 18:52:16,244::task::595::Storage.TaskManager.Task::(_updateState) Task=`15c0a08d-dab4-471b-95a7-513ef09685c7`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:16,244::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:16,244::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:16,245::task::993::Storage.TaskManager.Task::(_decref) Task=`15c0a08d-dab4-471b-95a7-513ef09685c7`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:16,250::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:16,250::task::595::Storage.TaskManager.Task::(_updateState) Task=`7326105d-0035-4e28-adc1-93c92943b84f`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:16,250::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-19::INFO::2014-07-14 18:52:16,251::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-19::DEBUG::2014-07-14 18:52:16,251::task::1191::Storage.TaskManager.Task::(prepare) Task=`7326105d-0035-4e28-adc1-93c92943b84f`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-19::DEBUG::2014-07-14 18:52:16,251::task::595::Storage.TaskManager.Task::(_updateState) Task=`7326105d-0035-4e28-adc1-93c92943b84f`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:16,251::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:16,251::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:16,252::task::993::Storage.TaskManager.Task::(_decref) Task=`7326105d-0035-4e28-adc1-93c92943b84f`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:16,270::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:16,270::task::595::Storage.TaskManager.Task::(_updateState) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:16,270::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None) >Thread-19::DEBUG::2014-07-14 18:52:16,271::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`58917b24-e4c1-4c6c-8b57-d5fcb1f6668b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '570' at 'spmStart' >Thread-19::DEBUG::2014-07-14 18:52:16,271::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'exclusive' >Thread-19::DEBUG::2014-07-14 18:52:16,271::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'exclusive' (1 active user) >Thread-19::DEBUG::2014-07-14 18:52:16,271::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`58917b24-e4c1-4c6c-8b57-d5fcb1f6668b`::Granted request >Thread-19::DEBUG::2014-07-14 18:52:16,273::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::_resourcesAcquired: Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c (exclusive) >Thread-19::DEBUG::2014-07-14 18:52:16,273::task::993::Storage.TaskManager.Task::(_decref) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::ref 1 aborting False >Thread-19::DEBUG::2014-07-14 18:52:16,273::taskManager::78::Storage.TaskManager::(scheduleJob) scheduled job spmStart for task b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::INFO::2014-07-14 18:52:16,273::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-19::DEBUG::2014-07-14 18:52:16,273::task::1183::Storage.TaskManager.Task::(prepare) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::Prepare: 1 jobs exist, move to acquiring >Thread-19::DEBUG::2014-07-14 18:52:16,273::task::595::Storage.TaskManager.Task::(_updateState) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::moving from state preparing -> state acquiring >Thread-19::DEBUG::2014-07-14 18:52:16,274::task::595::Storage.TaskManager.Task::(_updateState) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::moving from state acquiring -> state queued >Thread-19::DEBUG::2014-07-14 18:52:16,274::taskManager::56::Storage.TaskManager::(_queueTask) queuing task: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-6::DEBUG::2014-07-14 18:52:16,274::threadPool::57::Storage.ThreadPool::(setRunningTask) Number of running tasks: 1 >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:16,274::threadPool::200::Storage.ThreadPool.WorkerThread::(_processNextTask) Task: b753537e-04a1-4d45-9f06-2b98865fa552 running: <bound method Task.commit of <storage.task.Task instance at 0x1c4eb48>> with: None >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:16,274::task::1200::Storage.TaskManager.Task::(commit) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::committing task: b753537e-04a1-4d45-9f06-2b98865fa552 >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:16,275::task::595::Storage.TaskManager.Task::(_updateState) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::moving from state queued -> state running >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:16,275::task::905::Storage.TaskManager.Task::(_runJobs) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f3e986a6090>> (args: (-1, '-1', 250, 3) kwargs: {}) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:16,275::task::331::Storage.TaskManager.Task::(run) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f3e986a6090>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:16,275::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:16,275::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:52:16,274::taskManager::64::Storage.TaskManager::(_queueTask) task queued: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:16,276::task::1188::Storage.TaskManager.Task::(prepare) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::returning >Thread-19::DEBUG::2014-07-14 18:52:16,276::task::993::Storage.TaskManager.Task::(_decref) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::ref 1 aborting False >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:16,278::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be4cd943882c2df4da89ac9d42030969827cd416'] >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:16,279::misc::746::Storage.SamplingMethod::(__call__) Returning last result >b753537e-04a1-4d45-9f06-2b98865fa552::INFO::2014-07-14 18:52:16,279::clusterlock::183::Storage.SANLock::(acquireHostId) Acquiring host id for domain 79818395-80e1-40af-adb6-328223a20ad9 (id: 1) >Thread-19::DEBUG::2014-07-14 18:52:17,281::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:17,281::task::595::Storage.TaskManager.Task::(_updateState) Task=`d6400e9c-a284-443f-9c4b-5e12c8a54a97`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:17,282::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:17,282::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:17,282::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:17,282::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:17,282::task::1191::Storage.TaskManager.Task::(prepare) Task=`d6400e9c-a284-443f-9c4b-5e12c8a54a97`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:17,282::task::595::Storage.TaskManager.Task::(_updateState) Task=`d6400e9c-a284-443f-9c4b-5e12c8a54a97`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:17,283::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:17,283::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:17,283::task::993::Storage.TaskManager.Task::(_decref) Task=`d6400e9c-a284-443f-9c4b-5e12c8a54a97`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:18,287::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:18,287::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3363122-58ff-49c3-a87c-09d4e8600e98`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:18,288::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:18,288::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:18,288::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:18,288::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:18,288::task::1191::Storage.TaskManager.Task::(prepare) Task=`a3363122-58ff-49c3-a87c-09d4e8600e98`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:18,288::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3363122-58ff-49c3-a87c-09d4e8600e98`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:18,289::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:18,289::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:18,289::task::993::Storage.TaskManager.Task::(_decref) Task=`a3363122-58ff-49c3-a87c-09d4e8600e98`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:19,293::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:19,293::task::595::Storage.TaskManager.Task::(_updateState) Task=`efe4df2d-18cd-4454-be45-d2af34c08b2a`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:19,294::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:19,294::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:19,294::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:19,294::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:19,294::task::1191::Storage.TaskManager.Task::(prepare) Task=`efe4df2d-18cd-4454-be45-d2af34c08b2a`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:19,294::task::595::Storage.TaskManager.Task::(_updateState) Task=`efe4df2d-18cd-4454-be45-d2af34c08b2a`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:19,294::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:19,295::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:19,295::task::993::Storage.TaskManager.Task::(_decref) Task=`efe4df2d-18cd-4454-be45-d2af34c08b2a`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:20,299::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:20,299::task::595::Storage.TaskManager.Task::(_updateState) Task=`af739927-3cc4-4b5a-9a8f-e198ab728813`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:20,299::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:20,300::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:20,300::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:20,300::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:20,300::task::1191::Storage.TaskManager.Task::(prepare) Task=`af739927-3cc4-4b5a-9a8f-e198ab728813`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:20,300::task::595::Storage.TaskManager.Task::(_updateState) Task=`af739927-3cc4-4b5a-9a8f-e198ab728813`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:20,300::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:20,301::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:20,301::task::993::Storage.TaskManager.Task::(_decref) Task=`af739927-3cc4-4b5a-9a8f-e198ab728813`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:21,067::task::595::Storage.TaskManager.Task::(_updateState) Task=`d3e0017e-4127-4381-abe0-2cae3a4d410a`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:21,067::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:52:21,067::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000700515', 'lastCheck': '4.9', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:52:21,068::task::1191::Storage.TaskManager.Task::(prepare) Task=`d3e0017e-4127-4381-abe0-2cae3a4d410a`::finished: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000700515', 'lastCheck': '4.9', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:52:21,068::task::595::Storage.TaskManager.Task::(_updateState) Task=`d3e0017e-4127-4381-abe0-2cae3a4d410a`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:21,068::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:21,068::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:21,068::task::993::Storage.TaskManager.Task::(_decref) Task=`d3e0017e-4127-4381-abe0-2cae3a4d410a`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:21,305::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:21,305::task::595::Storage.TaskManager.Task::(_updateState) Task=`4dfc5481-e208-421a-9b88-792e24d6e11b`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:21,305::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:21,306::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:21,306::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:21,306::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:21,306::task::1191::Storage.TaskManager.Task::(prepare) Task=`4dfc5481-e208-421a-9b88-792e24d6e11b`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:21,306::task::595::Storage.TaskManager.Task::(_updateState) Task=`4dfc5481-e208-421a-9b88-792e24d6e11b`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:21,306::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:21,306::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:21,307::task::993::Storage.TaskManager.Task::(_decref) Task=`4dfc5481-e208-421a-9b88-792e24d6e11b`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:22,311::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:22,311::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b370345-4aec-4488-a476-7c3115b0047c`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:22,312::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:22,312::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:22,312::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:22,312::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:22,312::task::1191::Storage.TaskManager.Task::(prepare) Task=`0b370345-4aec-4488-a476-7c3115b0047c`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:22,312::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b370345-4aec-4488-a476-7c3115b0047c`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:22,313::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:22,313::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:22,313::task::993::Storage.TaskManager.Task::(_decref) Task=`0b370345-4aec-4488-a476-7c3115b0047c`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:23,317::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:23,317::task::595::Storage.TaskManager.Task::(_updateState) Task=`f47bb1fd-cd36-4c2c-9689-b8c16d9148d7`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:23,318::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:23,318::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:23,318::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:23,318::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:23,318::task::1191::Storage.TaskManager.Task::(prepare) Task=`f47bb1fd-cd36-4c2c-9689-b8c16d9148d7`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:23,318::task::595::Storage.TaskManager.Task::(_updateState) Task=`f47bb1fd-cd36-4c2c-9689-b8c16d9148d7`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:23,318::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:23,319::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:23,319::task::993::Storage.TaskManager.Task::(_decref) Task=`f47bb1fd-cd36-4c2c-9689-b8c16d9148d7`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:24,323::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:24,323::task::595::Storage.TaskManager.Task::(_updateState) Task=`73fb1913-e74a-4b75-a2a9-29f399dab845`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:24,323::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:24,324::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:24,324::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:24,324::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:24,324::task::1191::Storage.TaskManager.Task::(prepare) Task=`73fb1913-e74a-4b75-a2a9-29f399dab845`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:24,324::task::595::Storage.TaskManager.Task::(_updateState) Task=`73fb1913-e74a-4b75-a2a9-29f399dab845`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:24,324::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:24,324::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:24,325::task::993::Storage.TaskManager.Task::(_decref) Task=`73fb1913-e74a-4b75-a2a9-29f399dab845`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:25,329::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:25,329::task::595::Storage.TaskManager.Task::(_updateState) Task=`51ca0652-094d-4a3c-a95d-107f32527673`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:25,329::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:25,330::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:25,330::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:25,330::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:25,330::task::1191::Storage.TaskManager.Task::(prepare) Task=`51ca0652-094d-4a3c-a95d-107f32527673`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:25,330::task::595::Storage.TaskManager.Task::(_updateState) Task=`51ca0652-094d-4a3c-a95d-107f32527673`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:25,330::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:25,331::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:25,331::task::993::Storage.TaskManager.Task::(_decref) Task=`51ca0652-094d-4a3c-a95d-107f32527673`::ref 0 aborting False >Thread-46::DEBUG::2014-07-14 18:52:26,162::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-07-14 18:52:26,175::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n481 bytes (481 B) copied, 0.00063577 s, 757 kB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:26,335::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:26,336::task::595::Storage.TaskManager.Task::(_updateState) Task=`6771ea6f-b4bb-447b-83c2-e7f4c017c603`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:26,336::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:26,336::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:26,336::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:26,336::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:26,337::task::1191::Storage.TaskManager.Task::(prepare) Task=`6771ea6f-b4bb-447b-83c2-e7f4c017c603`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:26,337::task::595::Storage.TaskManager.Task::(_updateState) Task=`6771ea6f-b4bb-447b-83c2-e7f4c017c603`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:26,337::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:26,337::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:26,337::task::993::Storage.TaskManager.Task::(_decref) Task=`6771ea6f-b4bb-447b-83c2-e7f4c017c603`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:27,341::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:27,342::task::595::Storage.TaskManager.Task::(_updateState) Task=`7dcf7b55-82ab-4295-afa6-17985b55c8da`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:27,342::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:27,342::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:27,342::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:27,343::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:27,343::task::1191::Storage.TaskManager.Task::(prepare) Task=`7dcf7b55-82ab-4295-afa6-17985b55c8da`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:27,343::task::595::Storage.TaskManager.Task::(_updateState) Task=`7dcf7b55-82ab-4295-afa6-17985b55c8da`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:27,343::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:27,343::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:27,343::task::993::Storage.TaskManager.Task::(_decref) Task=`7dcf7b55-82ab-4295-afa6-17985b55c8da`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:28,348::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:28,349::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ccb9da2-a1ae-499d-92c9-af2bdcdcd67b`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:28,349::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:28,349::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:28,349::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:28,349::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:28,350::task::1191::Storage.TaskManager.Task::(prepare) Task=`3ccb9da2-a1ae-499d-92c9-af2bdcdcd67b`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:28,350::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ccb9da2-a1ae-499d-92c9-af2bdcdcd67b`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:28,350::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:28,350::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:28,350::task::993::Storage.TaskManager.Task::(_decref) Task=`3ccb9da2-a1ae-499d-92c9-af2bdcdcd67b`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:29,354::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:29,354::task::595::Storage.TaskManager.Task::(_updateState) Task=`166e26f3-834e-429d-a905-640bb746de13`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:29,355::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:29,355::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:29,355::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:29,355::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:29,355::task::1191::Storage.TaskManager.Task::(prepare) Task=`166e26f3-834e-429d-a905-640bb746de13`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:29,355::task::595::Storage.TaskManager.Task::(_updateState) Task=`166e26f3-834e-429d-a905-640bb746de13`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:29,356::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:29,356::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:29,356::task::993::Storage.TaskManager.Task::(_decref) Task=`166e26f3-834e-429d-a905-640bb746de13`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:30,360::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:30,360::task::595::Storage.TaskManager.Task::(_updateState) Task=`7e2f819a-31a6-4de2-8f8d-a3378ae2afde`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:30,360::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:30,361::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:30,361::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:30,361::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:30,361::task::1191::Storage.TaskManager.Task::(prepare) Task=`7e2f819a-31a6-4de2-8f8d-a3378ae2afde`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:30,361::task::595::Storage.TaskManager.Task::(_updateState) Task=`7e2f819a-31a6-4de2-8f8d-a3378ae2afde`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:30,361::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:30,361::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:30,362::task::993::Storage.TaskManager.Task::(_decref) Task=`7e2f819a-31a6-4de2-8f8d-a3378ae2afde`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:31,367::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:31,367::task::595::Storage.TaskManager.Task::(_updateState) Task=`28486710-1635-4b9c-b2bd-972cc499b4ec`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:31,368::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:31,368::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:31,368::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:31,368::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:31,368::task::1191::Storage.TaskManager.Task::(prepare) Task=`28486710-1635-4b9c-b2bd-972cc499b4ec`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:31,368::task::595::Storage.TaskManager.Task::(_updateState) Task=`28486710-1635-4b9c-b2bd-972cc499b4ec`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:31,369::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:31,369::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:31,369::task::993::Storage.TaskManager.Task::(_decref) Task=`28486710-1635-4b9c-b2bd-972cc499b4ec`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:32,373::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:32,373::task::595::Storage.TaskManager.Task::(_updateState) Task=`2be833b7-1605-426d-8814-b5927b1cff04`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:32,374::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:32,374::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:32,374::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:32,374::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:32,374::task::1191::Storage.TaskManager.Task::(prepare) Task=`2be833b7-1605-426d-8814-b5927b1cff04`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:32,375::task::595::Storage.TaskManager.Task::(_updateState) Task=`2be833b7-1605-426d-8814-b5927b1cff04`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:32,375::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:32,375::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:32,375::task::993::Storage.TaskManager.Task::(_decref) Task=`2be833b7-1605-426d-8814-b5927b1cff04`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:33,379::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:33,379::task::595::Storage.TaskManager.Task::(_updateState) Task=`14a3a03a-791b-48d2-8052-1974a6b9cd94`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:33,380::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:33,380::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:33,380::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:33,380::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:33,380::task::1191::Storage.TaskManager.Task::(prepare) Task=`14a3a03a-791b-48d2-8052-1974a6b9cd94`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:33,380::task::595::Storage.TaskManager.Task::(_updateState) Task=`14a3a03a-791b-48d2-8052-1974a6b9cd94`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:33,381::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:33,381::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:33,381::task::993::Storage.TaskManager.Task::(_decref) Task=`14a3a03a-791b-48d2-8052-1974a6b9cd94`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:34,385::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:34,385::task::595::Storage.TaskManager.Task::(_updateState) Task=`6f391c03-7941-4b65-b49a-52d31439a5b7`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:34,385::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:34,386::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:34,386::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:34,386::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:34,386::task::1191::Storage.TaskManager.Task::(prepare) Task=`6f391c03-7941-4b65-b49a-52d31439a5b7`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:34,386::task::595::Storage.TaskManager.Task::(_updateState) Task=`6f391c03-7941-4b65-b49a-52d31439a5b7`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:34,386::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:34,386::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:34,387::task::993::Storage.TaskManager.Task::(_decref) Task=`6f391c03-7941-4b65-b49a-52d31439a5b7`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:35,391::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:35,391::task::595::Storage.TaskManager.Task::(_updateState) Task=`ea49f2ca-8d45-4290-b151-57712ee77b58`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:35,391::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:35,391::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:35,392::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:35,392::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:35,392::task::1191::Storage.TaskManager.Task::(prepare) Task=`ea49f2ca-8d45-4290-b151-57712ee77b58`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:35,392::task::595::Storage.TaskManager.Task::(_updateState) Task=`ea49f2ca-8d45-4290-b151-57712ee77b58`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:35,392::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:35,392::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:35,392::task::993::Storage.TaskManager.Task::(_decref) Task=`ea49f2ca-8d45-4290-b151-57712ee77b58`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:36,178::task::595::Storage.TaskManager.Task::(_updateState) Task=`bde7eb2f-c0a5-457e-aff7-b7dbb9f0153c`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:36,178::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:52:36,178::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000700515', 'lastCheck': '20.0', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:52:36,179::task::1191::Storage.TaskManager.Task::(prepare) Task=`bde7eb2f-c0a5-457e-aff7-b7dbb9f0153c`::finished: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000700515', 'lastCheck': '20.0', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:52:36,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`bde7eb2f-c0a5-457e-aff7-b7dbb9f0153c`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:36,179::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:36,179::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:36,179::task::993::Storage.TaskManager.Task::(_decref) Task=`bde7eb2f-c0a5-457e-aff7-b7dbb9f0153c`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:36,397::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:36,397::task::595::Storage.TaskManager.Task::(_updateState) Task=`217e2207-f2e1-428d-a1db-9ce79ad6f05b`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:36,397::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:36,397::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:36,398::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:36,398::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:36,398::task::1191::Storage.TaskManager.Task::(prepare) Task=`217e2207-f2e1-428d-a1db-9ce79ad6f05b`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:36,398::task::595::Storage.TaskManager.Task::(_updateState) Task=`217e2207-f2e1-428d-a1db-9ce79ad6f05b`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:36,398::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:36,398::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:36,398::task::993::Storage.TaskManager.Task::(_decref) Task=`217e2207-f2e1-428d-a1db-9ce79ad6f05b`::ref 0 aborting False >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,281::clusterlock::201::Storage.SANLock::(acquireHostId) Host id for domain 79818395-80e1-40af-adb6-328223a20ad9 successfully acquired (id: 1) >b753537e-04a1-4d45-9f06-2b98865fa552::INFO::2014-07-14 18:52:37,282::clusterlock::234::Storage.SANLock::(acquire) Acquiring cluster lock for domain 79818395-80e1-40af-adb6-328223a20ad9 (id: 1) >Thread-19::DEBUG::2014-07-14 18:52:37,410::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:37,410::task::595::Storage.TaskManager.Task::(_updateState) Task=`21825589-002f-4ae3-8fad-68625aa5c786`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:37,411::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:37,411::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:37,411::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:37,411::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:37,411::task::1191::Storage.TaskManager.Task::(prepare) Task=`21825589-002f-4ae3-8fad-68625aa5c786`::finished: {'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:37,411::task::595::Storage.TaskManager.Task::(_updateState) Task=`21825589-002f-4ae3-8fad-68625aa5c786`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:37,411::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:37,412::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:37,412::task::993::Storage.TaskManager.Task::(_decref) Task=`21825589-002f-4ae3-8fad-68625aa5c786`::ref 0 aborting False >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,562::clusterlock::260::Storage.SANLock::(acquire) Cluster lock for domain 79818395-80e1-40af-adb6-328223a20ad9 successfully acquired (id: 1) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,562::sp::272::Storage.StoragePool::(startSpm) spm lock acquired successfully >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,565::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be4cd943882c2df4da89ac9d42030969827cd416'] >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,566::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,566::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,566::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=e42a6ad70b5711573490bf7db63e0d373d4d868f'] >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,570::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,571::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.upgrade_68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`207fa4d2-ab4d-4fb9-a693-c03fe6027a82`::Request was made in '/usr/share/vdsm/storage/sp.py' line '426' at '_upgradePool' >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,571::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'exclusive' >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,571::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.upgrade_68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'exclusive' (1 active user) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,571::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.upgrade_68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`207fa4d2-ab4d-4fb9-a693-c03fe6027a82`::Granted request >b753537e-04a1-4d45-9f06-2b98865fa552::INFO::2014-07-14 18:52:37,572::sp::432::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `79818395-80e1-40af-adb6-328223a20ad9` >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,572::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`1682891c-1f25-4dbf-88c3-bbbcbc1c48b9`::Request was made in '/usr/share/vdsm/storage/sp.py' line '434' at '_upgradePool' >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,572::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' for lock type 'exclusive' >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,573::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free. Now locking as 'exclusive' (1 active user) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,573::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`1682891c-1f25-4dbf-88c3-bbbcbc1c48b9`::Granted request >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,589::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,590::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,593::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=e42a6ad70b5711573490bf7db63e0d373d4d868f'] >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,595::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >b753537e-04a1-4d45-9f06-2b98865fa552::INFO::2014-07-14 18:52:37,595::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 79818395-80e1-40af-adb6-328223a20ad9_imageNS already registered >b753537e-04a1-4d45-9f06-2b98865fa552::INFO::2014-07-14 18:52:37,595::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 79818395-80e1-40af-adb6-328223a20ad9_volumeNS already registered >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,596::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,596::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' (0 active users) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,596::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free, finding out if anyone is waiting for it. >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,596::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9', Clearing records. >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,596::sp::437::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,599::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=e42a6ad70b5711573490bf7db63e0d373d4d868f'] >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,600::sp::444::Storage.StoragePool::(_upgradePool) Registering with state change event >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,600::sp::447::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,600::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_68055dba-2ea1-4ff1-864e-7c15d4b1334c' >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,600::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.upgrade_68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,600::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.upgrade_68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,601::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,601::fileSD::582::Storage.StorageDomain::(mountMaster) Creating master directory: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/master >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,603::sd::632::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/master/vms >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,604::sd::635::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/master/tasks >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,609::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=e42a6ad70b5711573490bf7db63e0d373d4d868f'] >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,613::threadPool::35::Storage.ThreadPool::(__init__) Enter - numThreads: 5, waitTimeout: 3, maxTasks: 500 >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,616::storage_mailbox::574::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor - clearing outgoing mail, command is: ['dd', 'of=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/outbox', 'oflag=direct', 'iflag=fullblock', 'conv=notrunc', 'count=1'] >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,616::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) 'dd of=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/outbox oflag=direct iflag=fullblock conv=notrunc count=1 bs=1024000' (cwd None) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,726::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0981611 s, 10.4 MB/s\n'; <rc> = 0 >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,727::storage_mailbox::582::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor created for pool 68055dba-2ea1-4ff1-864e-7c15d4b1334c >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,728::sp::319::Storage.StoragePool::(startSpm) SPM mailbox ready for pool 68055dba-2ea1-4ff1-864e-7c15d4b1334c on master domain 79818395-80e1-40af-adb6-328223a20ad9 >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,729::sp::327::Storage.StoragePool::(startSpm) ended. >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,730::task::595::Storage.TaskManager.Task::(_updateState) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::moving from state running -> state finished >Dummy-70::DEBUG::2014-07-14 18:52:37,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,731::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c': < ResourceRef 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', isValid: 'True' obj: 'None'>} >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,731::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,742::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,743::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,743::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,744::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,744::task::916::Storage.TaskManager.Task::(_runJobs) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::Task.run: exit - success: result >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,744::task::993::Storage.TaskManager.Task::(_decref) Task=`b753537e-04a1-4d45-9f06-2b98865fa552`::ref 0 aborting False >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:37,744::threadPool::57::Storage.ThreadPool::(setRunningTask) Number of running tasks: 0 >Dummy-70::DEBUG::2014-07-14 18:52:37,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926389 s, 11.1 MB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:38,416::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:38,417::task::595::Storage.TaskManager.Task::(_updateState) Task=`bff7769c-b6d6-4066-81b3-a9a6efd692b8`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:38,417::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:38,417::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:38,417::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'} >Thread-19::INFO::2014-07-14 18:52:38,417::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:38,417::task::1191::Storage.TaskManager.Task::(prepare) Task=`bff7769c-b6d6-4066-81b3-a9a6efd692b8`::finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b753537e-04a1-4d45-9f06-2b98865fa552'}} >Thread-19::DEBUG::2014-07-14 18:52:38,418::task::595::Storage.TaskManager.Task::(_updateState) Task=`bff7769c-b6d6-4066-81b3-a9a6efd692b8`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:38,418::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:38,418::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:38,418::task::993::Storage.TaskManager.Task::(_decref) Task=`bff7769c-b6d6-4066-81b3-a9a6efd692b8`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:38,422::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:38,422::task::595::Storage.TaskManager.Task::(_updateState) Task=`4258c1db-e524-4a1a-a3ce-4e22feeb90e5`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:38,422::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-19::INFO::2014-07-14 18:52:38,423::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:52:38,423::task::1191::Storage.TaskManager.Task::(prepare) Task=`4258c1db-e524-4a1a-a3ce-4e22feeb90e5`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:52:38,423::task::595::Storage.TaskManager.Task::(_updateState) Task=`4258c1db-e524-4a1a-a3ce-4e22feeb90e5`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:38,423::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:38,423::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:38,424::task::993::Storage.TaskManager.Task::(_decref) Task=`4258c1db-e524-4a1a-a3ce-4e22feeb90e5`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:38,429::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-19::DEBUG::2014-07-14 18:52:38,429::task::595::Storage.TaskManager.Task::(_updateState) Task=`8a777040-543f-4916-9216-aaa4d1d77b00`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:38,429::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='b753537e-04a1-4d45-9f06-2b98865fa552', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:38,429::taskManager::171::Storage.TaskManager::(clearTask) Entry. taskID: b753537e-04a1-4d45-9f06-2b98865fa552 >Thread-19::DEBUG::2014-07-14 18:52:38,430::taskManager::176::Storage.TaskManager::(clearTask) Return. >Thread-19::INFO::2014-07-14 18:52:38,430::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-19::DEBUG::2014-07-14 18:52:38,430::task::1191::Storage.TaskManager.Task::(prepare) Task=`8a777040-543f-4916-9216-aaa4d1d77b00`::finished: None >Thread-19::DEBUG::2014-07-14 18:52:38,430::task::595::Storage.TaskManager.Task::(_updateState) Task=`8a777040-543f-4916-9216-aaa4d1d77b00`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:38,430::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:38,430::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:38,430::task::993::Storage.TaskManager.Task::(_decref) Task=`8a777040-543f-4916-9216-aaa4d1d77b00`::ref 0 aborting False >Detector thread::DEBUG::2014-07-14 18:52:38,455::protocoldetector::159::vds.MultiProtocolAcceptor::(_add_connection) Adding connection from 10.35.4.70:52927 >Detector thread::DEBUG::2014-07-14 18:52:38,455::protocoldetector::170::vds.MultiProtocolAcceptor::(_remove_connection) Connection removed from 10.35.4.70:52927 >Detector thread::DEBUG::2014-07-14 18:52:38,455::protocoldetector::196::vds.MultiProtocolAcceptor::(_handle_connection_read) Detected protocol xml from 10.35.4.70:52927 >Detector thread::DEBUG::2014-07-14 18:52:38,456::BindingXMLRPC::1147::XmlDetector::(handleSocket) xml over http detected from ('10.35.4.70', 52927) >Thread-71::DEBUG::2014-07-14 18:52:38,458::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [65935655] >Thread-71::DEBUG::2014-07-14 18:52:38,458::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1e4dfdd-faf1-4824-b7bd-920c6d234e9b`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:52:38,458::logUtils::44::dispatcher::(wrapper) Run and protect: activateStorageDomain(sdUUID='79818395-80e1-40af-adb6-328223a20ad9', spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-71::DEBUG::2014-07-14 18:52:38,459::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`bd4846eb-efd8-4896-8a06-411a8f8ff43c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1219' at 'activateStorageDomain' >Thread-71::DEBUG::2014-07-14 18:52:38,459::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'exclusive' >Thread-71::DEBUG::2014-07-14 18:52:38,459::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'exclusive' (1 active user) >Thread-71::DEBUG::2014-07-14 18:52:38,459::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`bd4846eb-efd8-4896-8a06-411a8f8ff43c`::Granted request >Thread-71::DEBUG::2014-07-14 18:52:38,460::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a1e4dfdd-faf1-4824-b7bd-920c6d234e9b`::_resourcesAcquired: Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c (exclusive) >Thread-71::DEBUG::2014-07-14 18:52:38,460::task::993::Storage.TaskManager.Task::(_decref) Task=`a1e4dfdd-faf1-4824-b7bd-920c6d234e9b`::ref 1 aborting False >Thread-71::DEBUG::2014-07-14 18:52:38,460::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`654f08e5-1c88-4c05-9266-86c4a033f8f6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1220' at 'activateStorageDomain' >Thread-71::DEBUG::2014-07-14 18:52:38,460::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' for lock type 'exclusive' >Thread-71::DEBUG::2014-07-14 18:52:38,461::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free. Now locking as 'exclusive' (1 active user) >Thread-71::DEBUG::2014-07-14 18:52:38,461::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`654f08e5-1c88-4c05-9266-86c4a033f8f6`::Granted request >Thread-71::DEBUG::2014-07-14 18:52:38,461::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a1e4dfdd-faf1-4824-b7bd-920c6d234e9b`::_resourcesAcquired: Storage.79818395-80e1-40af-adb6-328223a20ad9 (exclusive) >Thread-71::DEBUG::2014-07-14 18:52:38,461::task::993::Storage.TaskManager.Task::(_decref) Task=`a1e4dfdd-faf1-4824-b7bd-920c6d234e9b`::ref 1 aborting False >Thread-71::INFO::2014-07-14 18:52:38,461::sp::1031::Storage.StoragePool::(activateSD) sdUUID=79818395-80e1-40af-adb6-328223a20ad9 spUUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c >Thread-71::INFO::2014-07-14 18:52:38,462::logUtils::47::dispatcher::(wrapper) Run and protect: activateStorageDomain, Return response: None >Thread-71::DEBUG::2014-07-14 18:52:38,462::task::1191::Storage.TaskManager.Task::(prepare) Task=`a1e4dfdd-faf1-4824-b7bd-920c6d234e9b`::finished: None >Thread-71::DEBUG::2014-07-14 18:52:38,462::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1e4dfdd-faf1-4824-b7bd-920c6d234e9b`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-07-14 18:52:38,462::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c': < ResourceRef 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', isValid: 'True' obj: 'None'>, 'Storage.79818395-80e1-40af-adb6-328223a20ad9': < ResourceRef 'Storage.79818395-80e1-40af-adb6-328223a20ad9', isValid: 'True' obj: 'None'>} >Thread-71::DEBUG::2014-07-14 18:52:38,462::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-07-14 18:52:38,463::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-71::DEBUG::2014-07-14 18:52:38,463::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-71::DEBUG::2014-07-14 18:52:38,463::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:52:38,463::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-71::DEBUG::2014-07-14 18:52:38,463::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' >Thread-71::DEBUG::2014-07-14 18:52:38,463::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' (0 active users) >Thread-71::DEBUG::2014-07-14 18:52:38,464::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:52:38,464::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9', Clearing records. >Thread-71::DEBUG::2014-07-14 18:52:38,464::task::993::Storage.TaskManager.Task::(_decref) Task=`a1e4dfdd-faf1-4824-b7bd-920c6d234e9b`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:38,486::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-19::DEBUG::2014-07-14 18:52:38,486::task::595::Storage.TaskManager.Task::(_updateState) Task=`56e12034-6719-4283-8785-e18d2175cc80`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:38,486::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', hostID=1, msdUUID='79818395-80e1-40af-adb6-328223a20ad9', masterVersion=1, domainsMap=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:38,487::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`75453c7b-7144-441c-b455-f4aff4522ffe`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '981' at 'connectStoragePool' >Thread-19::DEBUG::2014-07-14 18:52:38,487::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-19::DEBUG::2014-07-14 18:52:38,487::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-19::DEBUG::2014-07-14 18:52:38,487::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`75453c7b-7144-441c-b455-f4aff4522ffe`::Granted request >Thread-19::DEBUG::2014-07-14 18:52:38,488::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`10845cea-cc8a-4921-a97e-043c31fce9bd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1019' at '_connectStoragePool' >Thread-19::DEBUG::2014-07-14 18:52:38,488::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'shared' >Thread-19::DEBUG::2014-07-14 18:52:38,488::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'shared' (1 active user) >Thread-19::DEBUG::2014-07-14 18:52:38,488::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`10845cea-cc8a-4921-a97e-043c31fce9bd`::Granted request >Thread-19::DEBUG::2014-07-14 18:52:38,489::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,489::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,489::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,489::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,490::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,490::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,490::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-19::DEBUG::2014-07-14 18:52:38,490::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:52:38,490::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-19::DEBUG::2014-07-14 18:52:38,490::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:52:38,491::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-19::DEBUG::2014-07-14 18:52:38,491::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R' (cwd None) >Thread-19::DEBUG::2014-07-14 18:52:38,509::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:52:38,510::multipath::109::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /usr/sbin/multipath' (cwd None) >Thread-19::DEBUG::2014-07-14 18:52:38,543::multipath::109::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:38,543::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,543::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,544::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,544::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,544::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,544::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,544::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:52:38,563::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:38,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-19::DEBUG::2014-07-14 18:52:38,568::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=e42a6ad70b5711573490bf7db63e0d373d4d868f'] >Thread-19::DEBUG::2014-07-14 18:52:38,569::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-19::INFO::2014-07-14 18:52:38,569::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 79818395-80e1-40af-adb6-328223a20ad9_imageNS already registered >Thread-19::INFO::2014-07-14 18:52:38,569::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 79818395-80e1-40af-adb6-328223a20ad9_volumeNS already registered >Thread-19::DEBUG::2014-07-14 18:52:38,570::sp::1387::Storage.StoragePool::(setMasterDomain) Master domain 79818395-80e1-40af-adb6-328223a20ad9 verified, version 1 >Thread-19::DEBUG::2014-07-14 18:52:38,570::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-19::DEBUG::2014-07-14 18:52:38,570::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:52:38,570::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:52:38,570::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c >Thread-19::WARNING::2014-07-14 18:52:38,571::fileUtils::149::Storage.fileUtils::(createdir) Dir /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c already exists >Thread-19::DEBUG::2014-07-14 18:52:38,571::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:52:38,572::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-19::DEBUG::2014-07-14 18:52:38,599::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n No volume groups found\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:38,600::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19::INFO::2014-07-14 18:52:38,611::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::DEBUG::2014-07-14 18:52:38,612::sp::1135::Storage.StoragePool::(_linkStorageDomain) Creating symlink from /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/79818395-80e1-40af-adb6-328223a20ad9 >Thread-19::INFO::2014-07-14 18:52:38,612::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd >Thread-19::DEBUG::2014-07-14 18:52:38,612::sp::1135::Storage.StoragePool::(_linkStorageDomain) Creating symlink from /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9 to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd >Thread-19::DEBUG::2014-07-14 18:52:38,613::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-19::DEBUG::2014-07-14 18:52:38,613::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-19::DEBUG::2014-07-14 18:52:38,613::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:52:38,613::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-19::DEBUG::2014-07-14 18:52:38,614::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-19::DEBUG::2014-07-14 18:52:38,614::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-19::DEBUG::2014-07-14 18:52:38,614::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:52:38,614::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-19::INFO::2014-07-14 18:52:38,614::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-19::DEBUG::2014-07-14 18:52:38,614::task::1191::Storage.TaskManager.Task::(prepare) Task=`56e12034-6719-4283-8785-e18d2175cc80`::finished: True >Thread-19::DEBUG::2014-07-14 18:52:38,615::task::595::Storage.TaskManager.Task::(_updateState) Task=`56e12034-6719-4283-8785-e18d2175cc80`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:38,615::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:38,615::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:38,615::task::993::Storage.TaskManager.Task::(_decref) Task=`56e12034-6719-4283-8785-e18d2175cc80`::ref 0 aborting False >Dummy-70::DEBUG::2014-07-14 18:52:39,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:52:39,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916143 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:52:41,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:52:42,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091593 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:52:44,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:52:44,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916224 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:52:46,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-46::DEBUG::2014-07-14 18:52:47,598::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-07-14 18:52:47,611::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n479 bytes (479 B) copied, 0.000703728 s, 681 kB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:52:47,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.39641 s, 733 kB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:48,471::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-19::DEBUG::2014-07-14 18:52:48,472::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f64e822-4dfb-43ab-ba92-b67c751d66a2`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:48,472::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-19::INFO::2014-07-14 18:52:48,472::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:52:48,472::task::1191::Storage.TaskManager.Task::(prepare) Task=`7f64e822-4dfb-43ab-ba92-b67c751d66a2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:52:48,473::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f64e822-4dfb-43ab-ba92-b67c751d66a2`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:48,473::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:48,473::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:48,473::task::993::Storage.TaskManager.Task::(_decref) Task=`7f64e822-4dfb-43ab-ba92-b67c751d66a2`::ref 0 aborting False >Thread-71::DEBUG::2014-07-14 18:52:48,481::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-71::DEBUG::2014-07-14 18:52:48,481::task::595::Storage.TaskManager.Task::(_updateState) Task=`4bc3cc56-7245-45b6-8747-52f6047a9d8a`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:52:48,482::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-71::DEBUG::2014-07-14 18:52:48,482::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`0fbf2446-ef7f-43b2-9f75-c86f6ddec557`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2576' at 'getStoragePoolInfo' >Thread-71::DEBUG::2014-07-14 18:52:48,482::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'shared' >Thread-71::DEBUG::2014-07-14 18:52:48,482::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'shared' (1 active user) >Thread-71::DEBUG::2014-07-14 18:52:48,483::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`0fbf2446-ef7f-43b2-9f75-c86f6ddec557`::Granted request >Thread-71::DEBUG::2014-07-14 18:52:48,483::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4bc3cc56-7245-45b6-8747-52f6047a9d8a`::_resourcesAcquired: Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c (shared) >Thread-71::DEBUG::2014-07-14 18:52:48,483::task::993::Storage.TaskManager.Task::(_decref) Task=`4bc3cc56-7245-45b6-8747-52f6047a9d8a`::ref 1 aborting False >Thread-71::INFO::2014-07-14 18:52:48,491::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': '79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:52:48,491::task::1191::Storage.TaskManager.Task::(prepare) Task=`4bc3cc56-7245-45b6-8747-52f6047a9d8a`::finished: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': '79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:52:48,491::task::595::Storage.TaskManager.Task::(_updateState) Task=`4bc3cc56-7245-45b6-8747-52f6047a9d8a`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-07-14 18:52:48,491::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c': < ResourceRef 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', isValid: 'True' obj: 'None'>} >Thread-71::DEBUG::2014-07-14 18:52:48,492::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-07-14 18:52:48,492::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-71::DEBUG::2014-07-14 18:52:48,492::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-71::DEBUG::2014-07-14 18:52:48,492::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:52:48,492::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-71::DEBUG::2014-07-14 18:52:48,492::task::993::Storage.TaskManager.Task::(_decref) Task=`4bc3cc56-7245-45b6-8747-52f6047a9d8a`::ref 0 aborting False >Thread-71::DEBUG::2014-07-14 18:52:49,546::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [3df43e91] >Thread-71::DEBUG::2014-07-14 18:52:49,547::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:52:49,547::logUtils::44::dispatcher::(wrapper) Run and protect: createVolume(sdUUID='79818395-80e1-40af-adb6-328223a20ad9', spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', imgUUID='57708ce1-c83a-44d4-914c-c4d9c48fdde6', size='1073741824', volFormat=5, preallocate=2, diskType=2, volUUID='a2a75a21-c4ee-4781-82c4-9aa534036fff', desc='', srcImgUUID='00000000-0000-0000-0000-000000000000', srcVolUUID='00000000-0000-0000-0000-000000000000') >Thread-71::DEBUG::2014-07-14 18:52:49,547::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`a1a63261-36c4-4c0a-8538-a6131f7ad33c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1427' at 'createVolume' >Thread-71::DEBUG::2014-07-14 18:52:49,548::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' for lock type 'shared' >Thread-71::DEBUG::2014-07-14 18:52:49,548::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free. Now locking as 'shared' (1 active user) >Thread-71::DEBUG::2014-07-14 18:52:49,548::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`a1a63261-36c4-4c0a-8538-a6131f7ad33c`::Granted request >Thread-71::DEBUG::2014-07-14 18:52:49,548::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_resourcesAcquired: Storage.79818395-80e1-40af-adb6-328223a20ad9 (shared) >Thread-71::DEBUG::2014-07-14 18:52:49,549::task::993::Storage.TaskManager.Task::(_decref) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::ref 1 aborting False >Thread-71::DEBUG::2014-07-14 18:52:49,551::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >Thread-71::DEBUG::2014-07-14 18:52:49,562::taskManager::78::Storage.TaskManager::(scheduleJob) scheduled job createVolume for task c4b23c1c-6195-428f-b03f-e308c110800e >Thread-71::INFO::2014-07-14 18:52:49,563::logUtils::47::dispatcher::(wrapper) Run and protect: createVolume, Return response: None >Thread-71::DEBUG::2014-07-14 18:52:49,563::task::1183::Storage.TaskManager.Task::(prepare) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::Prepare: 1 jobs exist, move to acquiring >Thread-71::DEBUG::2014-07-14 18:52:49,563::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::moving from state preparing -> state acquiring >Thread-71::DEBUG::2014-07-14 18:52:49,563::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >Thread-71::DEBUG::2014-07-14 18:52:49,577::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::moving from state acquiring -> state queued >Thread-71::DEBUG::2014-07-14 18:52:49,578::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >Thread-71::DEBUG::2014-07-14 18:52:49,592::taskManager::56::Storage.TaskManager::(_queueTask) queuing task: c4b23c1c-6195-428f-b03f-e308c110800e >Thread-71::DEBUG::2014-07-14 18:52:49,592::taskManager::64::Storage.TaskManager::(_queueTask) task queued: c4b23c1c-6195-428f-b03f-e308c110800e >b753537e-04a1-4d45-9f06-2b98865fa552::DEBUG::2014-07-14 18:52:49,593::threadPool::57::Storage.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-71::DEBUG::2014-07-14 18:52:49,593::task::1188::Storage.TaskManager.Task::(prepare) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::returning >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,594::threadPool::200::Storage.ThreadPool.WorkerThread::(_processNextTask) Task: c4b23c1c-6195-428f-b03f-e308c110800e running: <bound method Task.commit of <storage.task.Task instance at 0x1905d88>> with: None >Thread-71::DEBUG::2014-07-14 18:52:49,594::task::993::Storage.TaskManager.Task::(_decref) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::ref 0 aborting False >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,595::task::1200::Storage.TaskManager.Task::(commit) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::committing task: c4b23c1c-6195-428f-b03f-e308c110800e >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,595::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::moving from state queued -> state running >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,597::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,611::task::905::Storage.TaskManager.Task::(_runJobs) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::Task.run: running job 0: createVolume: <bound method StoragePool.createVolume of <storage.sp.StoragePool object at 0x7f3e986a6090>> (args: ('79818395-80e1-40af-adb6-328223a20ad9', '57708ce1-c83a-44d4-914c-c4d9c48fdde6', 2097152, 5, 2, 2, 'a2a75a21-c4ee-4781-82c4-9aa534036fff', '', '00000000-0000-0000-0000-000000000000', '00000000-0000-0000-0000-000000000000') kwargs: {}) >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,612::task::331::Storage.TaskManager.Task::(run) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::Job.run: running createVolume: <bound method StoragePool.createVolume of <storage.sp.StoragePool object at 0x7f3e986a6090>> (args: ('79818395-80e1-40af-adb6-328223a20ad9', '57708ce1-c83a-44d4-914c-c4d9c48fdde6', 2097152, 5, 2, 2, 'a2a75a21-c4ee-4781-82c4-9aa534036fff', '', '00000000-0000-0000-0000-000000000000', '00000000-0000-0000-0000-000000000000') kwargs: {}) callback None >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,612::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`79818395-80e1-40af-adb6-328223a20ad9_imageNS.57708ce1-c83a-44d4-914c-c4d9c48fdde6`ReqID=`44744e10-ba1b-4d5d-ae46-abb82d5e03ed`::Request was made in '/usr/share/vdsm/storage/sp.py' line '1792' at 'createVolume' >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,612::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource '79818395-80e1-40af-adb6-328223a20ad9_imageNS.57708ce1-c83a-44d4-914c-c4d9c48fdde6' for lock type 'exclusive' >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,613::resourceFactories::125::Storage.ResourcesFactories::(__getResourceCandidatesList) Image 57708ce1-c83a-44d4-914c-c4d9c48fdde6 does not exist in domain 79818395-80e1-40af-adb6-328223a20ad9 >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,614::resourceManager::601::Storage.ResourceManager::(registerResource) Resource '79818395-80e1-40af-adb6-328223a20ad9_imageNS.57708ce1-c83a-44d4-914c-c4d9c48fdde6' is free. Now locking as 'exclusive' (1 active user) >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,614::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`79818395-80e1-40af-adb6-328223a20ad9_imageNS.57708ce1-c83a-44d4-914c-c4d9c48fdde6`ReqID=`44744e10-ba1b-4d5d-ae46-abb82d5e03ed`::Granted request >c4b23c1c-6195-428f-b03f-e308c110800e::INFO::2014-07-14 18:52:49,615::image::121::Storage.Image::(create) Create placeholder /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/79818395-80e1-40af-adb6-328223a20ad9/images/57708ce1-c83a-44d4-914c-c4d9c48fdde6 for image's volumes >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,615::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >Dummy-70::DEBUG::2014-07-14 18:52:49,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >c4b23c1c-6195-428f-b03f-e308c110800e::INFO::2014-07-14 18:52:49,733::volume::429::Storage.Volume::(create) Creating volume a2a75a21-c4ee-4781-82c4-9aa534036fff >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,735::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >Dummy-70::DEBUG::2014-07-14 18:52:49,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0950048 s, 10.8 MB/s\n'; <rc> = 0 >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,766::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >c4b23c1c-6195-428f-b03f-e308c110800e::INFO::2014-07-14 18:52:49,793::fileVolume::133::Storage.Volume::(_create) Request to create RAW volume /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/79818395-80e1-40af-adb6-328223a20ad9/images/57708ce1-c83a-44d4-914c-c4d9c48fdde6/a2a75a21-c4ee-4781-82c4-9aa534036fff with size = 2097152 sectors >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,795::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,824::fileVolume::384::Storage.Volume::(newVolumeLease) Initializing volume lease volUUID=a2a75a21-c4ee-4781-82c4-9aa534036fff sdUUID=79818395-80e1-40af-adb6-328223a20ad9, metaId=('/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/79818395-80e1-40af-adb6-328223a20ad9/images/57708ce1-c83a-44d4-914c-c4d9c48fdde6/a2a75a21-c4ee-4781-82c4-9aa534036fff',) >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,927::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,951::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource '79818395-80e1-40af-adb6-328223a20ad9_imageNS.57708ce1-c83a-44d4-914c-c4d9c48fdde6' >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,951::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource '79818395-80e1-40af-adb6-328223a20ad9_imageNS.57708ce1-c83a-44d4-914c-c4d9c48fdde6' (0 active users) >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,951::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource '79818395-80e1-40af-adb6-328223a20ad9_imageNS.57708ce1-c83a-44d4-914c-c4d9c48fdde6' is free, finding out if anyone is waiting for it. >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,951::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource '79818395-80e1-40af-adb6-328223a20ad9_imageNS.57708ce1-c83a-44d4-914c-c4d9c48fdde6', Clearing records. >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,952::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::moving from state running -> state finished >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,952::task::752::Storage.TaskManager.Task::(_save) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::_save: orig /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e temp /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/master/tasks/c4b23c1c-6195-428f-b03f-e308c110800e.temp >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,975::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.79818395-80e1-40af-adb6-328223a20ad9': < ResourceRef 'Storage.79818395-80e1-40af-adb6-328223a20ad9', isValid: 'True' obj: 'None'>} >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,975::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,975::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,976::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' (0 active users) >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,976::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free, finding out if anyone is waiting for it. >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,976::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9', Clearing records. >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,976::task::916::Storage.TaskManager.Task::(_runJobs) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::Task.run: exit - success: result {'uuid': 'a2a75a21-c4ee-4781-82c4-9aa534036fff'} >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,976::task::993::Storage.TaskManager.Task::(_decref) Task=`c4b23c1c-6195-428f-b03f-e308c110800e`::ref 0 aborting False >c4b23c1c-6195-428f-b03f-e308c110800e::DEBUG::2014-07-14 18:52:49,976::threadPool::57::Storage.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-19::DEBUG::2014-07-14 18:52:51,280::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fa50f94-8a7a-47ad-b91c-1778525b3faa`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:51,280::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:52:51,281::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000703728', 'lastCheck': '3.7', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:52:51,281::task::1191::Storage.TaskManager.Task::(prepare) Task=`1fa50f94-8a7a-47ad-b91c-1778525b3faa`::finished: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000703728', 'lastCheck': '3.7', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:52:51,281::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fa50f94-8a7a-47ad-b91c-1778525b3faa`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:51,281::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:51,281::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:51,282::task::993::Storage.TaskManager.Task::(_decref) Task=`1fa50f94-8a7a-47ad-b91c-1778525b3faa`::ref 0 aborting False >Dummy-70::DEBUG::2014-07-14 18:52:51,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:52:51,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0914041 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:52:53,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:52:53,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915438 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:52:55,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:52:56,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916666 s, 11.2 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-07-14 18:52:57,618::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-07-14 18:52:57,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n479 bytes (479 B) copied, 0.00063515 s, 754 kB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:57,890::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-19::DEBUG::2014-07-14 18:52:57,891::task::595::Storage.TaskManager.Task::(_updateState) Task=`9245b982-97d3-4ea1-8896-35cb50581100`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:57,891::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:57,891::taskManager::112::Storage.TaskManager::(getAllTasksStatuses) Entry. >Thread-19::DEBUG::2014-07-14 18:52:57,891::taskManager::103::Storage.TaskManager::(getTaskStatus) Entry. taskID: c4b23c1c-6195-428f-b03f-e308c110800e >Thread-19::DEBUG::2014-07-14 18:52:57,891::taskManager::106::Storage.TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'c4b23c1c-6195-428f-b03f-e308c110800e'} >Thread-19::DEBUG::2014-07-14 18:52:57,892::taskManager::123::Storage.TaskManager::(getAllTasksStatuses) Return: {'c4b23c1c-6195-428f-b03f-e308c110800e': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'c4b23c1c-6195-428f-b03f-e308c110800e'}} >Thread-19::INFO::2014-07-14 18:52:57,892::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {'c4b23c1c-6195-428f-b03f-e308c110800e': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'c4b23c1c-6195-428f-b03f-e308c110800e'}}} >Thread-19::DEBUG::2014-07-14 18:52:57,892::task::1191::Storage.TaskManager.Task::(prepare) Task=`9245b982-97d3-4ea1-8896-35cb50581100`::finished: {'allTasksStatus': {'c4b23c1c-6195-428f-b03f-e308c110800e': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'c4b23c1c-6195-428f-b03f-e308c110800e'}}} >Thread-19::DEBUG::2014-07-14 18:52:57,892::task::595::Storage.TaskManager.Task::(_updateState) Task=`9245b982-97d3-4ea1-8896-35cb50581100`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:57,892::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:57,892::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:57,893::task::993::Storage.TaskManager.Task::(_decref) Task=`9245b982-97d3-4ea1-8896-35cb50581100`::ref 0 aborting False >Thread-71::DEBUG::2014-07-14 18:52:57,952::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [3df43e91] >Thread-71::DEBUG::2014-07-14 18:52:57,953::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4c5d922-86e1-4533-a9f3-f3a52f1e7669`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:52:57,953::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeInfo(sdUUID='79818395-80e1-40af-adb6-328223a20ad9', spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', imgUUID='57708ce1-c83a-44d4-914c-c4d9c48fdde6', volUUID='a2a75a21-c4ee-4781-82c4-9aa534036fff', options=None) >Thread-71::DEBUG::2014-07-14 18:52:57,954::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`47f21fc3-730d-43fb-9b74-c1a01abda57b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3093' at 'getVolumeInfo' >Thread-71::DEBUG::2014-07-14 18:52:57,954::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' for lock type 'shared' >Thread-71::DEBUG::2014-07-14 18:52:57,954::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free. Now locking as 'shared' (1 active user) >Thread-71::DEBUG::2014-07-14 18:52:57,954::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.79818395-80e1-40af-adb6-328223a20ad9`ReqID=`47f21fc3-730d-43fb-9b74-c1a01abda57b`::Granted request >Thread-71::DEBUG::2014-07-14 18:52:57,955::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e4c5d922-86e1-4533-a9f3-f3a52f1e7669`::_resourcesAcquired: Storage.79818395-80e1-40af-adb6-328223a20ad9 (shared) >Thread-71::DEBUG::2014-07-14 18:52:57,955::task::993::Storage.TaskManager.Task::(_decref) Task=`e4c5d922-86e1-4533-a9f3-f3a52f1e7669`::ref 1 aborting False >Thread-71::DEBUG::2014-07-14 18:52:57,955::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for a2a75a21-c4ee-4781-82c4-9aa534036fff >Thread-71::INFO::2014-07-14 18:52:57,957::volume::862::Storage.Volume::(getInfo) Info request: sdUUID=79818395-80e1-40af-adb6-328223a20ad9 imgUUID=57708ce1-c83a-44d4-914c-c4d9c48fdde6 volUUID = a2a75a21-c4ee-4781-82c4-9aa534036fff >Thread-71::INFO::2014-07-14 18:52:57,966::volume::890::Storage.Volume::(getInfo) 79818395-80e1-40af-adb6-328223a20ad9/57708ce1-c83a-44d4-914c-c4d9c48fdde6/a2a75a21-c4ee-4781-82c4-9aa534036fff info is {'status': 'OK', 'domain': '79818395-80e1-40af-adb6-328223a20ad9', 'voltype': 'LEAF', 'description': '', 'parent': '00000000-0000-0000-0000-000000000000', 'format': 'RAW', 'image': '57708ce1-c83a-44d4-914c-c4d9c48fdde6', 'ctime': '1405353169', 'disktype': '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '1073741824', 'children': [], 'pool': '', 'capacity': '1073741824', 'uuid': 'a2a75a21-c4ee-4781-82c4-9aa534036fff', 'truesize': '512', 'type': 'SPARSE'} >Thread-71::INFO::2014-07-14 18:52:57,966::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeInfo, Return response: {'info': {'status': 'OK', 'domain': '79818395-80e1-40af-adb6-328223a20ad9', 'voltype': 'LEAF', 'description': '', 'parent': '00000000-0000-0000-0000-000000000000', 'format': 'RAW', 'image': '57708ce1-c83a-44d4-914c-c4d9c48fdde6', 'ctime': '1405353169', 'disktype': '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '1073741824', 'children': [], 'pool': '', 'capacity': '1073741824', 'uuid': 'a2a75a21-c4ee-4781-82c4-9aa534036fff', 'truesize': '512', 'type': 'SPARSE'}} >Thread-71::DEBUG::2014-07-14 18:52:57,966::task::1191::Storage.TaskManager.Task::(prepare) Task=`e4c5d922-86e1-4533-a9f3-f3a52f1e7669`::finished: {'info': {'status': 'OK', 'domain': '79818395-80e1-40af-adb6-328223a20ad9', 'voltype': 'LEAF', 'description': '', 'parent': '00000000-0000-0000-0000-000000000000', 'format': 'RAW', 'image': '57708ce1-c83a-44d4-914c-c4d9c48fdde6', 'ctime': '1405353169', 'disktype': '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '1073741824', 'children': [], 'pool': '', 'capacity': '1073741824', 'uuid': 'a2a75a21-c4ee-4781-82c4-9aa534036fff', 'truesize': '512', 'type': 'SPARSE'}} >Thread-71::DEBUG::2014-07-14 18:52:57,966::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4c5d922-86e1-4533-a9f3-f3a52f1e7669`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-07-14 18:52:57,966::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.79818395-80e1-40af-adb6-328223a20ad9': < ResourceRef 'Storage.79818395-80e1-40af-adb6-328223a20ad9', isValid: 'True' obj: 'None'>} >Thread-71::DEBUG::2014-07-14 18:52:57,967::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-07-14 18:52:57,967::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' >Thread-71::DEBUG::2014-07-14 18:52:57,967::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' (0 active users) >Thread-71::DEBUG::2014-07-14 18:52:57,967::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:52:57,967::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.79818395-80e1-40af-adb6-328223a20ad9', Clearing records. >Thread-71::DEBUG::2014-07-14 18:52:57,968::task::993::Storage.TaskManager.Task::(_decref) Task=`e4c5d922-86e1-4533-a9f3-f3a52f1e7669`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:52:58,000::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-19::DEBUG::2014-07-14 18:52:58,000::task::595::Storage.TaskManager.Task::(_updateState) Task=`5754d2d4-b676-403b-ba55-24a8dd5e41a1`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:58,001::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='c4b23c1c-6195-428f-b03f-e308c110800e', spUUID=None, options=None) >Thread-19::DEBUG::2014-07-14 18:52:58,001::taskManager::171::Storage.TaskManager::(clearTask) Entry. taskID: c4b23c1c-6195-428f-b03f-e308c110800e >Thread-19::DEBUG::2014-07-14 18:52:58,008::taskManager::176::Storage.TaskManager::(clearTask) Return. >Thread-19::INFO::2014-07-14 18:52:58,008::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-19::DEBUG::2014-07-14 18:52:58,008::task::1191::Storage.TaskManager.Task::(prepare) Task=`5754d2d4-b676-403b-ba55-24a8dd5e41a1`::finished: None >Thread-19::DEBUG::2014-07-14 18:52:58,008::task::595::Storage.TaskManager.Task::(_updateState) Task=`5754d2d4-b676-403b-ba55-24a8dd5e41a1`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:58,008::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:58,009::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:58,009::task::993::Storage.TaskManager.Task::(_decref) Task=`5754d2d4-b676-403b-ba55-24a8dd5e41a1`::ref 0 aborting False >Dummy-70::DEBUG::2014-07-14 18:52:58,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:52:58,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926827 s, 11.0 MB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:52:58,510::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-19::DEBUG::2014-07-14 18:52:58,510::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ab65b2f-7146-4cbd-b1c6-bb7f70b5ff31`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:52:58,511::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-19::INFO::2014-07-14 18:52:58,511::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:52:58,511::task::1191::Storage.TaskManager.Task::(prepare) Task=`5ab65b2f-7146-4cbd-b1c6-bb7f70b5ff31`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:52:58,511::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ab65b2f-7146-4cbd-b1c6-bb7f70b5ff31`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:52:58,512::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:52:58,512::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:52:58,512::task::993::Storage.TaskManager.Task::(_decref) Task=`5ab65b2f-7146-4cbd-b1c6-bb7f70b5ff31`::ref 0 aborting False >Thread-71::DEBUG::2014-07-14 18:52:58,520::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-71::DEBUG::2014-07-14 18:52:58,520::task::595::Storage.TaskManager.Task::(_updateState) Task=`b7d1afc1-60a7-40e1-bfc0-1f2b909d05e8`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:52:58,520::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-71::DEBUG::2014-07-14 18:52:58,521::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`76d64495-1084-4ab6-9d44-5893750638f6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2576' at 'getStoragePoolInfo' >Thread-71::DEBUG::2014-07-14 18:52:58,521::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'shared' >Thread-71::DEBUG::2014-07-14 18:52:58,521::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'shared' (1 active user) >Thread-71::DEBUG::2014-07-14 18:52:58,521::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`76d64495-1084-4ab6-9d44-5893750638f6`::Granted request >Thread-71::DEBUG::2014-07-14 18:52:58,522::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b7d1afc1-60a7-40e1-bfc0-1f2b909d05e8`::_resourcesAcquired: Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c (shared) >Thread-71::DEBUG::2014-07-14 18:52:58,522::task::993::Storage.TaskManager.Task::(_decref) Task=`b7d1afc1-60a7-40e1-bfc0-1f2b909d05e8`::ref 1 aborting False >Thread-71::INFO::2014-07-14 18:52:58,530::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': '79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:52:58,530::task::1191::Storage.TaskManager.Task::(prepare) Task=`b7d1afc1-60a7-40e1-bfc0-1f2b909d05e8`::finished: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': '79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:52:58,530::task::595::Storage.TaskManager.Task::(_updateState) Task=`b7d1afc1-60a7-40e1-bfc0-1f2b909d05e8`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-07-14 18:52:58,530::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c': < ResourceRef 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', isValid: 'True' obj: 'None'>} >Thread-71::DEBUG::2014-07-14 18:52:58,530::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-07-14 18:52:58,530::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-71::DEBUG::2014-07-14 18:52:58,531::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-71::DEBUG::2014-07-14 18:52:58,531::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:52:58,531::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-71::DEBUG::2014-07-14 18:52:58,531::task::993::Storage.TaskManager.Task::(_decref) Task=`b7d1afc1-60a7-40e1-bfc0-1f2b909d05e8`::ref 0 aborting False >Dummy-70::DEBUG::2014-07-14 18:53:00,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:00,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916492 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:02,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:02,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0920806 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:04,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:04,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917038 s, 11.2 MB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:53:06,434::task::595::Storage.TaskManager.Task::(_updateState) Task=`5721d1ac-c2f2-4714-ba42-a47201176b51`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:06,434::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:53:06,435::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00063515', 'lastCheck': '8.8', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:53:06,435::task::1191::Storage.TaskManager.Task::(prepare) Task=`5721d1ac-c2f2-4714-ba42-a47201176b51`::finished: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00063515', 'lastCheck': '8.8', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:53:06,435::task::595::Storage.TaskManager.Task::(_updateState) Task=`5721d1ac-c2f2-4714-ba42-a47201176b51`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:06,435::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:06,435::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:06,435::task::993::Storage.TaskManager.Task::(_decref) Task=`5721d1ac-c2f2-4714-ba42-a47201176b51`::ref 0 aborting False >Dummy-70::DEBUG::2014-07-14 18:53:06,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:06,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964868 s, 10.6 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-07-14 18:53:07,635::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-07-14 18:53:07,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n479 bytes (479 B) copied, 0.000670645 s, 714 kB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:53:08,543::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-19::DEBUG::2014-07-14 18:53:08,544::task::595::Storage.TaskManager.Task::(_updateState) Task=`d1596de8-8804-47c5-bf4f-61f74bcedc6d`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:08,544::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-19::INFO::2014-07-14 18:53:08,545::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:53:08,545::task::1191::Storage.TaskManager.Task::(prepare) Task=`d1596de8-8804-47c5-bf4f-61f74bcedc6d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:53:08,545::task::595::Storage.TaskManager.Task::(_updateState) Task=`d1596de8-8804-47c5-bf4f-61f74bcedc6d`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:08,545::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:08,545::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:08,545::task::993::Storage.TaskManager.Task::(_decref) Task=`d1596de8-8804-47c5-bf4f-61f74bcedc6d`::ref 0 aborting False >Thread-71::DEBUG::2014-07-14 18:53:08,552::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-71::DEBUG::2014-07-14 18:53:08,552::task::595::Storage.TaskManager.Task::(_updateState) Task=`2082a622-32aa-4a61-9f2e-d60bc4501608`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:53:08,552::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-71::DEBUG::2014-07-14 18:53:08,553::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`98d4fde3-fd54-4452-b385-dc85c9983c09`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2576' at 'getStoragePoolInfo' >Thread-71::DEBUG::2014-07-14 18:53:08,553::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'shared' >Thread-71::DEBUG::2014-07-14 18:53:08,553::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'shared' (1 active user) >Thread-71::DEBUG::2014-07-14 18:53:08,553::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`98d4fde3-fd54-4452-b385-dc85c9983c09`::Granted request >Thread-71::DEBUG::2014-07-14 18:53:08,554::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2082a622-32aa-4a61-9f2e-d60bc4501608`::_resourcesAcquired: Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c (shared) >Thread-71::DEBUG::2014-07-14 18:53:08,554::task::993::Storage.TaskManager.Task::(_decref) Task=`2082a622-32aa-4a61-9f2e-d60bc4501608`::ref 1 aborting False >Thread-71::INFO::2014-07-14 18:53:08,562::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': '79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:53:08,562::task::1191::Storage.TaskManager.Task::(prepare) Task=`2082a622-32aa-4a61-9f2e-d60bc4501608`::finished: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': '79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:53:08,562::task::595::Storage.TaskManager.Task::(_updateState) Task=`2082a622-32aa-4a61-9f2e-d60bc4501608`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-07-14 18:53:08,562::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c': < ResourceRef 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', isValid: 'True' obj: 'None'>} >Thread-71::DEBUG::2014-07-14 18:53:08,562::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-07-14 18:53:08,563::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-71::DEBUG::2014-07-14 18:53:08,563::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-71::DEBUG::2014-07-14 18:53:08,563::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:53:08,563::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-71::DEBUG::2014-07-14 18:53:08,563::task::993::Storage.TaskManager.Task::(_decref) Task=`2082a622-32aa-4a61-9f2e-d60bc4501608`::ref 0 aborting False >Dummy-70::DEBUG::2014-07-14 18:53:08,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:08,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916737 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:10,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:10,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916406 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:12,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:13,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919781 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:15,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:15,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0980187 s, 10.4 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:17,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-46::DEBUG::2014-07-14 18:53:17,817::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-07-14 18:53:17,830::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n479 bytes (479 B) copied, 0.000662925 s, 723 kB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:17,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.522743 s, 2.0 MB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:53:18,578::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-19::DEBUG::2014-07-14 18:53:18,579::task::595::Storage.TaskManager.Task::(_updateState) Task=`9854b303-add3-44a3-be2f-2bc603bab57b`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:18,579::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-19::INFO::2014-07-14 18:53:18,579::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:53:18,579::task::1191::Storage.TaskManager.Task::(prepare) Task=`9854b303-add3-44a3-be2f-2bc603bab57b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:53:18,580::task::595::Storage.TaskManager.Task::(_updateState) Task=`9854b303-add3-44a3-be2f-2bc603bab57b`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:18,580::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:18,580::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:18,580::task::993::Storage.TaskManager.Task::(_decref) Task=`9854b303-add3-44a3-be2f-2bc603bab57b`::ref 0 aborting False >Thread-71::DEBUG::2014-07-14 18:53:18,587::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-71::DEBUG::2014-07-14 18:53:18,587::task::595::Storage.TaskManager.Task::(_updateState) Task=`a55dd86b-dd25-4d41-b2e6-3da07d771ea0`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:53:18,587::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-71::DEBUG::2014-07-14 18:53:18,588::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`510dfc04-b583-4071-be7c-6bcb73604707`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2576' at 'getStoragePoolInfo' >Thread-71::DEBUG::2014-07-14 18:53:18,588::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'shared' >Thread-71::DEBUG::2014-07-14 18:53:18,588::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'shared' (1 active user) >Thread-71::DEBUG::2014-07-14 18:53:18,588::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`510dfc04-b583-4071-be7c-6bcb73604707`::Granted request >Thread-71::DEBUG::2014-07-14 18:53:18,589::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a55dd86b-dd25-4d41-b2e6-3da07d771ea0`::_resourcesAcquired: Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c (shared) >Thread-71::DEBUG::2014-07-14 18:53:18,589::task::993::Storage.TaskManager.Task::(_decref) Task=`a55dd86b-dd25-4d41-b2e6-3da07d771ea0`::ref 1 aborting False >Thread-71::INFO::2014-07-14 18:53:18,597::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': '79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:53:18,597::task::1191::Storage.TaskManager.Task::(prepare) Task=`a55dd86b-dd25-4d41-b2e6-3da07d771ea0`::finished: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': '79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:53:18,597::task::595::Storage.TaskManager.Task::(_updateState) Task=`a55dd86b-dd25-4d41-b2e6-3da07d771ea0`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-07-14 18:53:18,597::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c': < ResourceRef 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', isValid: 'True' obj: 'None'>} >Thread-71::DEBUG::2014-07-14 18:53:18,597::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-07-14 18:53:18,598::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-71::DEBUG::2014-07-14 18:53:18,598::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-71::DEBUG::2014-07-14 18:53:18,598::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:53:18,598::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-71::DEBUG::2014-07-14 18:53:18,598::task::993::Storage.TaskManager.Task::(_decref) Task=`a55dd86b-dd25-4d41-b2e6-3da07d771ea0`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:53:19,154::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [4bd5ac2f] >Thread-19::DEBUG::2014-07-14 18:53:19,154::task::595::Storage.TaskManager.Task::(_updateState) Task=`09eceff5-ee41-408e-bb27-81d26eae07e3`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:19,154::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:53:19,162::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3 >Thread-19::DEBUG::2014-07-14 18:53:19,162::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.35.16.25:/volumes/smamit/frontend/gchaplik/data3 /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:19,231::hsm::2332::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3 >Thread-19::DEBUG::2014-07-14 18:53:19,233::hsm::2356::Storage.HSM::(__prefetchDomains) Found SD uuids: () >Thread-19::DEBUG::2014-07-14 18:53:19,233::hsm::2412::Storage.HSM::(connectStorageServer) knownSDs: {79818395-80e1-40af-adb6-328223a20ad9: storage.nfsSD.findDomain} >Thread-19::INFO::2014-07-14 18:53:19,234::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-19::DEBUG::2014-07-14 18:53:19,234::task::1191::Storage.TaskManager.Task::(prepare) Task=`09eceff5-ee41-408e-bb27-81d26eae07e3`::finished: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-19::DEBUG::2014-07-14 18:53:19,234::task::595::Storage.TaskManager.Task::(_updateState) Task=`09eceff5-ee41-408e-bb27-81d26eae07e3`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:19,234::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:19,234::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:19,234::task::993::Storage.TaskManager.Task::(_decref) Task=`09eceff5-ee41-408e-bb27-81d26eae07e3`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:53:19,326::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [5f5087c0] >Thread-19::DEBUG::2014-07-14 18:53:19,326::task::595::Storage.TaskManager.Task::(_updateState) Task=`53e06c73-9032-48ec-a368-67e55c82ac6b`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:19,326::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '562abf05-0385-428e-b6e8-79b090698188', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:53:19,334::hsm::2332::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3 >Thread-19::DEBUG::2014-07-14 18:53:19,335::hsm::2356::Storage.HSM::(__prefetchDomains) Found SD uuids: () >Thread-19::DEBUG::2014-07-14 18:53:19,335::hsm::2412::Storage.HSM::(connectStorageServer) knownSDs: {79818395-80e1-40af-adb6-328223a20ad9: storage.nfsSD.findDomain} >Thread-19::INFO::2014-07-14 18:53:19,335::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '562abf05-0385-428e-b6e8-79b090698188'}]} >Thread-19::DEBUG::2014-07-14 18:53:19,335::task::1191::Storage.TaskManager.Task::(prepare) Task=`53e06c73-9032-48ec-a368-67e55c82ac6b`::finished: {'statuslist': [{'status': 0, 'id': '562abf05-0385-428e-b6e8-79b090698188'}]} >Thread-19::DEBUG::2014-07-14 18:53:19,336::task::595::Storage.TaskManager.Task::(_updateState) Task=`53e06c73-9032-48ec-a368-67e55c82ac6b`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:19,336::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:19,336::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:19,336::task::993::Storage.TaskManager.Task::(_decref) Task=`53e06c73-9032-48ec-a368-67e55c82ac6b`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:53:19,349::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [5f5087c0] >Thread-19::DEBUG::2014-07-14 18:53:19,349::task::595::Storage.TaskManager.Task::(_updateState) Task=`8cf867d3-7231-4ee3-8654-f44ef89cf073`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:19,349::logUtils::44::dispatcher::(wrapper) Run and protect: createStorageDomain(storageType=1, sdUUID='d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', domainName='data3', typeSpecificArg='10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', domClass=1, domVersion='3', options=None) >Thread-19::DEBUG::2014-07-14 18:53:19,350::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-19::DEBUG::2014-07-14 18:53:19,350::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:53:19,350::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-19::DEBUG::2014-07-14 18:53:19,350::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:53:19,350::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-19::DEBUG::2014-07-14 18:53:19,351::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:19,369::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:53:19,370::multipath::109::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /usr/sbin/multipath' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:19,402::multipath::109::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:53:19,403::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:19,403::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:19,404::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:19,404::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:19,404::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:19,404::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:19,404::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::ERROR::2014-07-14 18:53:19,405::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b >Thread-19::ERROR::2014-07-14 18:53:19,405::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b >Thread-19::DEBUG::2014-07-14 18:53:19,405::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:19,406::lvm::288::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /usr/sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:19,438::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n Volume group "d3b42b81-9524-42a9-9ebf-f54b5e9bef6b" not found\n Skipping volume group d3b42b81-9524-42a9-9ebf-f54b5e9bef6b\n'; <rc> = 5 >Thread-19::WARNING::2014-07-14 18:53:19,439::lvm::370::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', ' Volume group "d3b42b81-9524-42a9-9ebf-f54b5e9bef6b" not found', ' Skipping volume group d3b42b81-9524-42a9-9ebf-f54b5e9bef6b'] >Thread-19::DEBUG::2014-07-14 18:53:19,440::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19::ERROR::2014-07-14 18:53:19,553::sdc::143::Storage.StorageDomainCache::(_findDomain) domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('d3b42b81-9524-42a9-9ebf-f54b5e9bef6b',) >Thread-19::INFO::2014-07-14 18:53:19,553::nfsSD::69::Storage.StorageDomain::(create) sdUUID=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b domainName=data3 remotePath=10.35.16.25:/volumes/smamit/frontend/gchaplik/data3 domClass=1 >Thread-19::DEBUG::2014-07-14 18:53:19,685::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-19::DEBUG::2014-07-14 18:53:19,686::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-19::WARNING::2014-07-14 18:53:19,686::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-19::DEBUG::2014-07-14 18:53:19,686::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-19::DEBUG::2014-07-14 18:53:19,686::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-19::DEBUG::2014-07-14 18:53:19,686::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'ROLE=Regular', 'SDUUID=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=fbfb658ff51f13ef74fdc4bf8530234f4e5cfda7'] >Thread-19::DEBUG::2014-07-14 18:53:19,690::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-19::DEBUG::2014-07-14 18:53:19,691::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3/d3b42b81-9524-42a9-9ebf-f54b5e9bef6b >Thread-19::DEBUG::2014-07-14 18:53:19,692::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-19::DEBUG::2014-07-14 18:53:19,696::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'ROLE=Regular', 'SDUUID=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=fbfb658ff51f13ef74fdc4bf8530234f4e5cfda7'] >Thread-19::DEBUG::2014-07-14 18:53:19,699::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-19::DEBUG::2014-07-14 18:53:19,699::resourceManager::421::Storage.ResourceManager::(registerNamespace) Registering namespace 'd3b42b81-9524-42a9-9ebf-f54b5e9bef6b_imageNS' >Thread-19::DEBUG::2014-07-14 18:53:19,699::resourceManager::421::Storage.ResourceManager::(registerNamespace) Registering namespace 'd3b42b81-9524-42a9-9ebf-f54b5e9bef6b_volumeNS' >Thread-19::DEBUG::2014-07-14 18:53:19,699::clusterlock::143::Storage.initSANLock::(initSANLock) Initializing SANLock for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b >Dummy-70::DEBUG::2014-07-14 18:53:19,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:20,441::sd::433::Storage.StorageDomain::(initSPMlease) lease initialized successfully >Thread-19::DEBUG::2014-07-14 18:53:20,442::hsm::2651::Storage.HSM::(createStorageDomain) knownSDs: {d3b42b81-9524-42a9-9ebf-f54b5e9bef6b: storage.nfsSD.findDomain, 79818395-80e1-40af-adb6-328223a20ad9: storage.nfsSD.findDomain} >Thread-19::INFO::2014-07-14 18:53:20,442::logUtils::47::dispatcher::(wrapper) Run and protect: createStorageDomain, Return response: None >Thread-19::DEBUG::2014-07-14 18:53:20,443::task::1191::Storage.TaskManager.Task::(prepare) Task=`8cf867d3-7231-4ee3-8654-f44ef89cf073`::finished: None >Thread-19::DEBUG::2014-07-14 18:53:20,443::task::595::Storage.TaskManager.Task::(_updateState) Task=`8cf867d3-7231-4ee3-8654-f44ef89cf073`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:20,444::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:20,444::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:20,445::task::993::Storage.TaskManager.Task::(_decref) Task=`8cf867d3-7231-4ee3-8654-f44ef89cf073`::ref 0 aborting False >Dummy-70::DEBUG::2014-07-14 18:53:20,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.556457 s, 1.8 MB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:53:20,453::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [5f5087c0] >Thread-19::DEBUG::2014-07-14 18:53:20,454::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b037c36-f9b6-463a-a2d1-c71188fa5d31`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:20,455::logUtils::44::dispatcher::(wrapper) Run and protect: getStorageDomainStats(sdUUID='d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', options=None) >Thread-19::DEBUG::2014-07-14 18:53:20,457::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b`ReqID=`aca93c2a-3b3d-4756-b59d-5ce529fcb1ea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2790' at 'getStorageDomainStats' >Thread-19::DEBUG::2014-07-14 18:53:20,458::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' for lock type 'shared' >Thread-19::DEBUG::2014-07-14 18:53:20,459::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' is free. Now locking as 'shared' (1 active user) >Thread-19::DEBUG::2014-07-14 18:53:20,459::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b`ReqID=`aca93c2a-3b3d-4756-b59d-5ce529fcb1ea`::Granted request >Thread-19::DEBUG::2014-07-14 18:53:20,460::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3b037c36-f9b6-463a-a2d1-c71188fa5d31`::_resourcesAcquired: Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b (shared) >Thread-19::DEBUG::2014-07-14 18:53:20,461::task::993::Storage.TaskManager.Task::(_decref) Task=`3b037c36-f9b6-463a-a2d1-c71188fa5d31`::ref 1 aborting False >Thread-19::INFO::2014-07-14 18:53:20,463::logUtils::47::dispatcher::(wrapper) Run and protect: getStorageDomainStats, Return response: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '417153941504', 'disktotal': '536870912000', 'mdafree': 0}} >Thread-19::DEBUG::2014-07-14 18:53:20,464::task::1191::Storage.TaskManager.Task::(prepare) Task=`3b037c36-f9b6-463a-a2d1-c71188fa5d31`::finished: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '417153941504', 'disktotal': '536870912000', 'mdafree': 0}} >Thread-19::DEBUG::2014-07-14 18:53:20,465::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b037c36-f9b6-463a-a2d1-c71188fa5d31`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:20,465::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b': < ResourceRef 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', isValid: 'True' obj: 'None'>} >Thread-19::DEBUG::2014-07-14 18:53:20,466::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:20,467::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' >Thread-19::DEBUG::2014-07-14 18:53:20,467::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' (0 active users) >Thread-19::DEBUG::2014-07-14 18:53:20,468::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:53:20,468::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', Clearing records. >Thread-19::DEBUG::2014-07-14 18:53:20,469::task::993::Storage.TaskManager.Task::(_decref) Task=`3b037c36-f9b6-463a-a2d1-c71188fa5d31`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:53:20,571::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [10e8b2d0] >Thread-19::DEBUG::2014-07-14 18:53:20,572::task::595::Storage.TaskManager.Task::(_updateState) Task=`d0972e01-6f30-4b6f-a804-c0535e20ff58`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:20,572::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '562abf05-0385-428e-b6e8-79b090698188', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:53:20,572::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:20,608::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-19::DEBUG::2014-07-14 18:53:20,608::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:53:20,608::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-19::DEBUG::2014-07-14 18:53:20,608::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:53:20,609::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-19::DEBUG::2014-07-14 18:53:20,609::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:20,629::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:53:20,629::multipath::109::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /usr/sbin/multipath' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:20,686::multipath::109::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:53:20,686::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,687::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,687::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,687::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,687::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,687::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,688::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::INFO::2014-07-14 18:53:20,688::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '562abf05-0385-428e-b6e8-79b090698188'}]} >Thread-19::DEBUG::2014-07-14 18:53:20,688::task::1191::Storage.TaskManager.Task::(prepare) Task=`d0972e01-6f30-4b6f-a804-c0535e20ff58`::finished: {'statuslist': [{'status': 0, 'id': '562abf05-0385-428e-b6e8-79b090698188'}]} >Thread-19::DEBUG::2014-07-14 18:53:20,688::task::595::Storage.TaskManager.Task::(_updateState) Task=`d0972e01-6f30-4b6f-a804-c0535e20ff58`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:20,688::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:20,689::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:20,689::task::993::Storage.TaskManager.Task::(_decref) Task=`d0972e01-6f30-4b6f-a804-c0535e20ff58`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:53:20,807::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [33752eb4] >Thread-19::DEBUG::2014-07-14 18:53:20,808::task::595::Storage.TaskManager.Task::(_updateState) Task=`74896db8-486a-43e9-b9c9-657d00a03afd`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:20,808::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '562abf05-0385-428e-b6e8-79b090698188', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:53:20,816::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3 >Thread-19::DEBUG::2014-07-14 18:53:20,816::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.35.16.25:/volumes/smamit/frontend/gchaplik/data3 /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:20,882::hsm::2332::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3 >Thread-19::DEBUG::2014-07-14 18:53:20,883::hsm::2356::Storage.HSM::(__prefetchDomains) Found SD uuids: ('d3b42b81-9524-42a9-9ebf-f54b5e9bef6b',) >Thread-19::DEBUG::2014-07-14 18:53:20,884::hsm::2412::Storage.HSM::(connectStorageServer) knownSDs: {d3b42b81-9524-42a9-9ebf-f54b5e9bef6b: storage.nfsSD.findDomain, 79818395-80e1-40af-adb6-328223a20ad9: storage.nfsSD.findDomain} >Thread-19::INFO::2014-07-14 18:53:20,884::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '562abf05-0385-428e-b6e8-79b090698188'}]} >Thread-19::DEBUG::2014-07-14 18:53:20,884::task::1191::Storage.TaskManager.Task::(prepare) Task=`74896db8-486a-43e9-b9c9-657d00a03afd`::finished: {'statuslist': [{'status': 0, 'id': '562abf05-0385-428e-b6e8-79b090698188'}]} >Thread-19::DEBUG::2014-07-14 18:53:20,884::task::595::Storage.TaskManager.Task::(_updateState) Task=`74896db8-486a-43e9-b9c9-657d00a03afd`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:20,885::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:20,885::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:20,885::task::993::Storage.TaskManager.Task::(_decref) Task=`74896db8-486a-43e9-b9c9-657d00a03afd`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:53:20,895::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [1807ee42] >Thread-19::DEBUG::2014-07-14 18:53:20,895::task::595::Storage.TaskManager.Task::(_updateState) Task=`fff51d79-57be-4b1e-bc6f-8e13e6fe2ff6`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:20,896::logUtils::44::dispatcher::(wrapper) Run and protect: getStorageDomainInfo(sdUUID='d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', options=None) >Thread-19::DEBUG::2014-07-14 18:53:20,896::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-19::DEBUG::2014-07-14 18:53:20,896::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:53:20,896::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-19::DEBUG::2014-07-14 18:53:20,896::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-19::DEBUG::2014-07-14 18:53:20,896::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-19::DEBUG::2014-07-14 18:53:20,897::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:20,915::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:53:20,916::multipath::109::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /usr/sbin/multipath' (cwd None) >Thread-19::DEBUG::2014-07-14 18:53:20,944::multipath::109::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:53:20,945::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,945::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,945::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,945::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,945::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,946::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-19::DEBUG::2014-07-14 18:53:20,946::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-19::DEBUG::2014-07-14 18:53:20,966::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3/d3b42b81-9524-42a9-9ebf-f54b5e9bef6b >Thread-19::DEBUG::2014-07-14 18:53:20,968::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-19::DEBUG::2014-07-14 18:53:20,971::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'ROLE=Regular', 'SDUUID=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=fbfb658ff51f13ef74fdc4bf8530234f4e5cfda7'] >Thread-19::DEBUG::2014-07-14 18:53:20,973::fileSD::619::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-19::INFO::2014-07-14 18:53:20,974::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace d3b42b81-9524-42a9-9ebf-f54b5e9bef6b_imageNS already registered >Thread-19::INFO::2014-07-14 18:53:20,974::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace d3b42b81-9524-42a9-9ebf-f54b5e9bef6b_volumeNS already registered >Thread-19::INFO::2014-07-14 18:53:20,974::fileSD::350::Storage.StorageDomain::(validate) sdUUID=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b >Thread-19::DEBUG::2014-07-14 18:53:20,977::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'ROLE=Regular', 'SDUUID=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=fbfb658ff51f13ef74fdc4bf8530234f4e5cfda7'] >Thread-19::DEBUG::2014-07-14 18:53:20,978::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b`ReqID=`a6186450-caea-44a0-b0e7-8cd3a91138ee`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2772' at 'getStorageDomainInfo' >Thread-19::DEBUG::2014-07-14 18:53:20,978::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' for lock type 'shared' >Thread-19::DEBUG::2014-07-14 18:53:20,978::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' is free. Now locking as 'shared' (1 active user) >Thread-19::DEBUG::2014-07-14 18:53:20,978::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b`ReqID=`a6186450-caea-44a0-b0e7-8cd3a91138ee`::Granted request >Thread-19::DEBUG::2014-07-14 18:53:20,979::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`fff51d79-57be-4b1e-bc6f-8e13e6fe2ff6`::_resourcesAcquired: Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b (shared) >Thread-19::DEBUG::2014-07-14 18:53:20,979::task::993::Storage.TaskManager.Task::(_decref) Task=`fff51d79-57be-4b1e-bc6f-8e13e6fe2ff6`::ref 1 aborting False >Thread-19::INFO::2014-07-14 18:53:20,986::logUtils::47::dispatcher::(wrapper) Run and protect: getStorageDomainInfo, Return response: {'info': {'uuid': 'd3b42b81-9524-42a9-9ebf-f54b5e9bef6b', 'version': '3', 'role': 'Regular', 'remotePath': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'type': 'NFS', 'class': 'Data', 'pool': [], 'name': 'data3'}} >Thread-19::DEBUG::2014-07-14 18:53:20,987::task::1191::Storage.TaskManager.Task::(prepare) Task=`fff51d79-57be-4b1e-bc6f-8e13e6fe2ff6`::finished: {'info': {'uuid': 'd3b42b81-9524-42a9-9ebf-f54b5e9bef6b', 'version': '3', 'role': 'Regular', 'remotePath': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'type': 'NFS', 'class': 'Data', 'pool': [], 'name': 'data3'}} >Thread-19::DEBUG::2014-07-14 18:53:20,987::task::595::Storage.TaskManager.Task::(_updateState) Task=`fff51d79-57be-4b1e-bc6f-8e13e6fe2ff6`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:20,987::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b': < ResourceRef 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', isValid: 'True' obj: 'None'>} >Thread-19::DEBUG::2014-07-14 18:53:20,987::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:20,987::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' >Thread-19::DEBUG::2014-07-14 18:53:20,988::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' (0 active users) >Thread-19::DEBUG::2014-07-14 18:53:20,988::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' is free, finding out if anyone is waiting for it. >Thread-19::DEBUG::2014-07-14 18:53:20,988::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', Clearing records. >Thread-19::DEBUG::2014-07-14 18:53:20,988::task::993::Storage.TaskManager.Task::(_decref) Task=`fff51d79-57be-4b1e-bc6f-8e13e6fe2ff6`::ref 0 aborting False >Thread-71::DEBUG::2014-07-14 18:53:20,996::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [1807ee42] >Thread-71::DEBUG::2014-07-14 18:53:20,997::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e690309-6f49-493a-9c47-b398af726f34`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:53:20,997::logUtils::44::dispatcher::(wrapper) Run and protect: attachStorageDomain(sdUUID='d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-71::DEBUG::2014-07-14 18:53:20,997::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`305051fb-4d5f-4df2-9d60-fa11e38093ed`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1160' at 'attachStorageDomain' >Thread-71::DEBUG::2014-07-14 18:53:20,997::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'exclusive' >Thread-71::DEBUG::2014-07-14 18:53:20,998::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'exclusive' (1 active user) >Thread-71::DEBUG::2014-07-14 18:53:20,998::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`305051fb-4d5f-4df2-9d60-fa11e38093ed`::Granted request >Thread-71::DEBUG::2014-07-14 18:53:20,998::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3e690309-6f49-493a-9c47-b398af726f34`::_resourcesAcquired: Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c (exclusive) >Thread-71::DEBUG::2014-07-14 18:53:20,998::task::993::Storage.TaskManager.Task::(_decref) Task=`3e690309-6f49-493a-9c47-b398af726f34`::ref 1 aborting False >Thread-71::DEBUG::2014-07-14 18:53:20,999::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b`ReqID=`fcf5e101-5ad6-411a-97b5-52e449a59521`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1161' at 'attachStorageDomain' >Thread-71::DEBUG::2014-07-14 18:53:20,999::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' for lock type 'exclusive' >Thread-71::DEBUG::2014-07-14 18:53:20,999::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' is free. Now locking as 'exclusive' (1 active user) >Thread-71::DEBUG::2014-07-14 18:53:20,999::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b`ReqID=`fcf5e101-5ad6-411a-97b5-52e449a59521`::Granted request >Thread-71::DEBUG::2014-07-14 18:53:20,999::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3e690309-6f49-493a-9c47-b398af726f34`::_resourcesAcquired: Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b (exclusive) >Thread-71::DEBUG::2014-07-14 18:53:21,000::task::993::Storage.TaskManager.Task::(_decref) Task=`3e690309-6f49-493a-9c47-b398af726f34`::ref 1 aborting False >Thread-71::INFO::2014-07-14 18:53:21,000::sp::876::Storage.StoragePool::(attachSD) sdUUID=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b spUUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c >Thread-71::INFO::2014-07-14 18:53:21,008::clusterlock::183::Storage.SANLock::(acquireHostId) Acquiring host id for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b (id: 1) >Thread-19::DEBUG::2014-07-14 18:53:21,530::task::595::Storage.TaskManager.Task::(_updateState) Task=`c536af9b-3dae-41fc-8406-1aa73127a6b7`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:21,530::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:53:21,531::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000662925', 'lastCheck': '3.7', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:53:21,531::task::1191::Storage.TaskManager.Task::(prepare) Task=`c536af9b-3dae-41fc-8406-1aa73127a6b7`::finished: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000662925', 'lastCheck': '3.7', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:53:21,531::task::595::Storage.TaskManager.Task::(_updateState) Task=`c536af9b-3dae-41fc-8406-1aa73127a6b7`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:21,531::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:21,531::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:21,531::task::993::Storage.TaskManager.Task::(_decref) Task=`c536af9b-3dae-41fc-8406-1aa73127a6b7`::ref 0 aborting False >Dummy-70::DEBUG::2014-07-14 18:53:22,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:22,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916462 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:24,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:24,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918762 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:26,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:27,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.558632 s, 1.8 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-07-14 18:53:27,839::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-07-14 18:53:27,852::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n479 bytes (479 B) copied, 0.000647066 s, 740 kB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:29,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:29,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.545576 s, 1.9 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:31,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:31,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919602 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:33,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:34,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917844 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:36,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:36,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916922 s, 11.2 MB/s\n'; <rc> = 0 >Thread-19::DEBUG::2014-07-14 18:53:36,640::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c89da2b-7f3d-49cf-aa2d-bf14abef0497`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:36,640::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-19::INFO::2014-07-14 18:53:36,641::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647066', 'lastCheck': '8.8', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:53:36,641::task::1191::Storage.TaskManager.Task::(prepare) Task=`6c89da2b-7f3d-49cf-aa2d-bf14abef0497`::finished: {'79818395-80e1-40af-adb6-328223a20ad9': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000647066', 'lastCheck': '8.8', 'valid': True}} >Thread-19::DEBUG::2014-07-14 18:53:36,641::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c89da2b-7f3d-49cf-aa2d-bf14abef0497`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:36,641::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:36,641::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:36,641::task::993::Storage.TaskManager.Task::(_decref) Task=`6c89da2b-7f3d-49cf-aa2d-bf14abef0497`::ref 0 aborting False >Thread-46::DEBUG::2014-07-14 18:53:37,856::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data2/79818395-80e1-40af-adb6-328223a20ad9/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-07-14 18:53:37,869::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n479 bytes (479 B) copied, 0.000678498 s, 706 kB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:38,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:38,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916264 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-70::DEBUG::2014-07-14 18:53:40,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:40,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916203 s, 11.2 MB/s\n'; <rc> = 0 >Thread-71::DEBUG::2014-07-14 18:53:42,010::clusterlock::201::Storage.SANLock::(acquireHostId) Host id for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b successfully acquired (id: 1) >Thread-71::INFO::2014-07-14 18:53:42,011::clusterlock::234::Storage.SANLock::(acquire) Acquiring cluster lock for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b (id: 1) >Dummy-70::DEBUG::2014-07-14 18:53:42,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-71::DEBUG::2014-07-14 18:53:42,710::clusterlock::260::Storage.SANLock::(acquire) Cluster lock for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b successfully acquired (id: 1) >Thread-71::DEBUG::2014-07-14 18:53:42,801::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'ROLE=Regular', 'SDUUID=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=fbfb658ff51f13ef74fdc4bf8530234f4e5cfda7'] >Dummy-70::DEBUG::2014-07-14 18:53:42,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917434 s, 11.2 MB/s\n'; <rc> = 0 >Thread-71::DEBUG::2014-07-14 18:53:42,805::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-71::DEBUG::2014-07-14 18:53:42,807::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-71::DEBUG::2014-07-14 18:53:42,809::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'ROLE=Regular', 'SDUUID=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=177497013fab607069f776c9626ba3da4a96fd6a'] >Thread-71::DEBUG::2014-07-14 18:53:42,815::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-71::DEBUG::2014-07-14 18:53:42,817::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-71::DEBUG::2014-07-14 18:53:42,819::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-71::DEBUG::2014-07-14 18:53:42,820::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data2', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc34', 'POOL_DOMAINS=d3b42b81-9524-42a9-9ebf-f54b5e9bef6b:Attached,79818395-80e1-40af-adb6-328223a20ad9:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=68055dba-2ea1-4ff1-864e-7c15d4b1334c', 'REMOTE_PATH=10.35.16.25:/volumes/smamit/frontend/gchaplik/data2', 'ROLE=Master', 'SDUUID=79818395-80e1-40af-adb6-328223a20ad9', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b5b13ec7c92e966e3df71d5bb293fc7cf64000b6'] >Thread-71::DEBUG::2014-07-14 18:53:42,826::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-71::INFO::2014-07-14 18:53:42,827::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3/d3b42b81-9524-42a9-9ebf-f54b5e9bef6b to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/d3b42b81-9524-42a9-9ebf-f54b5e9bef6b >Thread-71::DEBUG::2014-07-14 18:53:42,828::sp::1135::Storage.StoragePool::(_linkStorageDomain) Creating symlink from /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3/d3b42b81-9524-42a9-9ebf-f54b5e9bef6b to /rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/d3b42b81-9524-42a9-9ebf-f54b5e9bef6b >Thread-71::INFO::2014-07-14 18:53:42,830::clusterlock::278::Storage.SANLock::(release) Releasing cluster lock for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b >Thread-71::DEBUG::2014-07-14 18:53:42,833::clusterlock::288::Storage.SANLock::(release) Cluster lock for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b successfully released >Thread-71::INFO::2014-07-14 18:53:42,834::clusterlock::206::Storage.SANLock::(releaseHostId) Releasing host id for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b (id: 1) >Dummy-70::DEBUG::2014-07-14 18:53:44,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/68055dba-2ea1-4ff1-864e-7c15d4b1334c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-70::DEBUG::2014-07-14 18:53:44,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101556 s, 10.1 MB/s\n'; <rc> = 0 >Thread-71::DEBUG::2014-07-14 18:53:45,835::clusterlock::216::Storage.SANLock::(releaseHostId) Host id for domain d3b42b81-9524-42a9-9ebf-f54b5e9bef6b released successfully (id: 1) >Thread-71::DEBUG::2014-07-14 18:53:45,835::misc::736::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-71::DEBUG::2014-07-14 18:53:45,835::misc::738::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-71::DEBUG::2014-07-14 18:53:45,835::misc::746::Storage.SamplingMethod::(__call__) Returning last result >Thread-71::INFO::2014-07-14 18:53:45,836::logUtils::47::dispatcher::(wrapper) Run and protect: attachStorageDomain, Return response: None >Thread-71::DEBUG::2014-07-14 18:53:45,836::task::1191::Storage.TaskManager.Task::(prepare) Task=`3e690309-6f49-493a-9c47-b398af726f34`::finished: None >Thread-71::DEBUG::2014-07-14 18:53:45,836::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e690309-6f49-493a-9c47-b398af726f34`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-07-14 18:53:45,836::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c': < ResourceRef 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', isValid: 'True' obj: 'None'>, 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b': < ResourceRef 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', isValid: 'True' obj: 'None'>} >Thread-71::DEBUG::2014-07-14 18:53:45,836::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-07-14 18:53:45,837::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-71::DEBUG::2014-07-14 18:53:45,837::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-71::DEBUG::2014-07-14 18:53:45,837::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:53:45,837::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-71::DEBUG::2014-07-14 18:53:45,837::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' >Thread-71::DEBUG::2014-07-14 18:53:45,837::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' (0 active users) >Thread-71::DEBUG::2014-07-14 18:53:45,838::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:53:45,838::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', Clearing records. >Thread-71::DEBUG::2014-07-14 18:53:45,838::task::993::Storage.TaskManager.Task::(_decref) Task=`3e690309-6f49-493a-9c47-b398af726f34`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:53:45,843::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-19::DEBUG::2014-07-14 18:53:45,844::task::595::Storage.TaskManager.Task::(_updateState) Task=`be4d69df-4cb2-41b6-9b44-49fde9406cc7`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:45,844::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-19::INFO::2014-07-14 18:53:45,844::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:53:45,845::task::1191::Storage.TaskManager.Task::(prepare) Task=`be4d69df-4cb2-41b6-9b44-49fde9406cc7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-19::DEBUG::2014-07-14 18:53:45,845::task::595::Storage.TaskManager.Task::(_updateState) Task=`be4d69df-4cb2-41b6-9b44-49fde9406cc7`::moving from state preparing -> state finished >Thread-19::DEBUG::2014-07-14 18:53:45,845::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-19::DEBUG::2014-07-14 18:53:45,845::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-19::DEBUG::2014-07-14 18:53:45,845::task::993::Storage.TaskManager.Task::(_decref) Task=`be4d69df-4cb2-41b6-9b44-49fde9406cc7`::ref 0 aborting False >Thread-71::DEBUG::2014-07-14 18:53:45,852::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] >Thread-71::DEBUG::2014-07-14 18:53:45,852::task::595::Storage.TaskManager.Task::(_updateState) Task=`9becaa7f-886e-4ac7-a45a-87003d880095`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:53:45,852::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='68055dba-2ea1-4ff1-864e-7c15d4b1334c', options=None) >Thread-71::DEBUG::2014-07-14 18:53:45,853::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`ef1fc62e-fad8-4f0d-995b-14adedf7e96f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2576' at 'getStoragePoolInfo' >Thread-71::DEBUG::2014-07-14 18:53:45,853::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' for lock type 'shared' >Thread-71::DEBUG::2014-07-14 18:53:45,853::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free. Now locking as 'shared' (1 active user) >Thread-71::DEBUG::2014-07-14 18:53:45,853::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c`ReqID=`ef1fc62e-fad8-4f0d-995b-14adedf7e96f`::Granted request >Thread-71::DEBUG::2014-07-14 18:53:45,854::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9becaa7f-886e-4ac7-a45a-87003d880095`::_resourcesAcquired: Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c (shared) >Thread-71::DEBUG::2014-07-14 18:53:45,854::task::993::Storage.TaskManager.Task::(_decref) Task=`9becaa7f-886e-4ac7-a45a-87003d880095`::ref 1 aborting False >Thread-71::INFO::2014-07-14 18:53:45,862::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': 'd3b42b81-9524-42a9-9ebf-f54b5e9bef6b:Attached,79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'d3b42b81-9524-42a9-9ebf-f54b5e9bef6b': {'status': 'Attached', 'isoprefix': '', 'alerts': []}, '79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:53:45,862::task::1191::Storage.TaskManager.Task::(prepare) Task=`9becaa7f-886e-4ac7-a45a-87003d880095`::finished: {'info': {'name': 'dc34', 'isoprefix': '', 'pool_status': 'connected', 'lver': 0, 'domains': 'd3b42b81-9524-42a9-9ebf-f54b5e9bef6b:Attached,79818395-80e1-40af-adb6-328223a20ad9:Active', 'master_uuid': '79818395-80e1-40af-adb6-328223a20ad9', 'version': '3', 'spm_id': 1, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'d3b42b81-9524-42a9-9ebf-f54b5e9bef6b': {'status': 'Attached', 'isoprefix': '', 'alerts': []}, '79818395-80e1-40af-adb6-328223a20ad9': {'status': 'Active', 'diskfree': '417153941504', 'isoprefix': '', 'alerts': [], 'disktotal': '536870912000', 'version': 3}}} >Thread-71::DEBUG::2014-07-14 18:53:45,862::task::595::Storage.TaskManager.Task::(_updateState) Task=`9becaa7f-886e-4ac7-a45a-87003d880095`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-07-14 18:53:45,863::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c': < ResourceRef 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', isValid: 'True' obj: 'None'>} >Thread-71::DEBUG::2014-07-14 18:53:45,863::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-07-14 18:53:45,863::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' >Thread-71::DEBUG::2014-07-14 18:53:45,863::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' (0 active users) >Thread-71::DEBUG::2014-07-14 18:53:45,863::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:53:45,863::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.68055dba-2ea1-4ff1-864e-7c15d4b1334c', Clearing records. >Thread-71::DEBUG::2014-07-14 18:53:45,864::task::993::Storage.TaskManager.Task::(_decref) Task=`9becaa7f-886e-4ac7-a45a-87003d880095`::ref 0 aborting False >Thread-71::DEBUG::2014-07-14 18:53:45,875::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [1807ee42] >Thread-71::DEBUG::2014-07-14 18:53:45,876::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c755413-cc50-4d90-9187-64b3be85cf4b`::moving from state init -> state preparing >Thread-71::INFO::2014-07-14 18:53:45,876::logUtils::44::dispatcher::(wrapper) Run and protect: getImagesList(sdUUID='d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', options=None) >Thread-71::DEBUG::2014-07-14 18:53:45,876::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b`ReqID=`0d415399-7966-4027-9378-a093849c5bb2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3260' at 'getImagesList' >Thread-71::DEBUG::2014-07-14 18:53:45,876::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' for lock type 'shared' >Thread-71::DEBUG::2014-07-14 18:53:45,877::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' is free. Now locking as 'shared' (1 active user) >Thread-71::DEBUG::2014-07-14 18:53:45,877::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b`ReqID=`0d415399-7966-4027-9378-a093849c5bb2`::Granted request >Thread-71::DEBUG::2014-07-14 18:53:45,877::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2c755413-cc50-4d90-9187-64b3be85cf4b`::_resourcesAcquired: Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b (shared) >Thread-71::DEBUG::2014-07-14 18:53:45,877::task::993::Storage.TaskManager.Task::(_decref) Task=`2c755413-cc50-4d90-9187-64b3be85cf4b`::ref 1 aborting False >Thread-71::INFO::2014-07-14 18:53:45,879::logUtils::47::dispatcher::(wrapper) Run and protect: getImagesList, Return response: {'imageslist': []} >Thread-71::DEBUG::2014-07-14 18:53:45,879::task::1191::Storage.TaskManager.Task::(prepare) Task=`2c755413-cc50-4d90-9187-64b3be85cf4b`::finished: {'imageslist': []} >Thread-71::DEBUG::2014-07-14 18:53:45,879::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c755413-cc50-4d90-9187-64b3be85cf4b`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-07-14 18:53:45,879::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b': < ResourceRef 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', isValid: 'True' obj: 'None'>} >Thread-71::DEBUG::2014-07-14 18:53:45,879::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-07-14 18:53:45,880::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' >Thread-71::DEBUG::2014-07-14 18:53:45,880::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' (0 active users) >Thread-71::DEBUG::2014-07-14 18:53:45,880::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b' is free, finding out if anyone is waiting for it. >Thread-71::DEBUG::2014-07-14 18:53:45,880::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.d3b42b81-9524-42a9-9ebf-f54b5e9bef6b', Clearing records. >Thread-71::DEBUG::2014-07-14 18:53:45,880::task::993::Storage.TaskManager.Task::(_decref) Task=`2c755413-cc50-4d90-9187-64b3be85cf4b`::ref 0 aborting False >Thread-19::DEBUG::2014-07-14 18:53:45,923::BindingXMLRPC::298::vds::(wrapper) client [10.35.4.70] flowID [1486610f] >Thread-19::DEBUG::2014-07-14 18:53:45,924::task::595::Storage.TaskManager.Task::(_updateState) Task=`ded3c889-f240-48d5-8f00-2021c5f93349`::moving from state init -> state preparing >Thread-19::INFO::2014-07-14 18:53:45,924::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.35.16.25:/volumes/smamit/frontend/gchaplik/data3', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '562abf05-0385-428e-b6e8-79b090698188', 'port': ''}], options=None) >Thread-19::DEBUG::2014-07-14 18:53:45,932::hsm::2332::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.35.16.25:_volumes_smamit_frontend_gchaplik_data3 >Thread-19::DEBUG::2014-07-14 18:53:45,933::hsm::2356::Storage.HSM::(__prefetchDomains) Found SD uuids: ('d3b42b81-9524-42a9-9ebf-f54b5e9bef6b',) >Thread-19::DEBUG::2014-07-14 18:53:45,933::hsm::2412::Storage.HSM::(connectStorageServer) knownSDs: {d3b42b81-9524-42a9-9ebf-f54b5e9bef6b: storage.nfsSD.findDomain, 79818395-80e1-40af-adb6-328223a20ad9: storage.nfsSD.findDomain} >Thread-19::INFO::2014-07-14 18:53:45,933::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '562abf05-0385-428e-b6e8-79b090698188'}]} >Thread-19::DEBUG::2014-07-14 18:53:45,933::task::1191::Storage.TaskManager.Task::(prepare) Task=`ded3c889-f240-48d5-8f00-2021c5f93349`::finished: {'statuslist': [{'status': 0, 'id': '562abf05-0385-428e-b6e8-79b090698188'}]} >